application-gateway-kubernetes-ingress
application-gateway-kubernetes-ingress copied to clipboard
Published
20 hours ago •
Azure
Azure
AGIC crashed at Applcation Gateway doesn't have a public IP
trafficstars
Describe the bug AGIC 1.5.1 crashed when working with internal Application Gateway, error message:
F0531 10:05:48.200347 1 main.go:192] Got a fatal validation error on existing Application Gateway config. Please update Application Gateway or the controller's helm config. Error:Code="ErrorNoPublicIP" Message="Applcation Gateway doesn't have a public IP"
To Reproduce Steps to reproduce the behavior:
-
Create internal Application Gateway: Follow Configure an application gateway with an internal load balancer (ILB) endpoint to provision Standard tier application gateway on subnet(10.3.0.0/28). The frontend IP is
10.3.0.6. -
Enable AGIC 1.5.1 See helm configuration:
# This file contains the essential configs for the ingress controller Helm chart
# Verbosity level of the App Gateway Ingress Controller
verbosityLevel: 3
################################################################################
# Specify which application gateway the ingress controller will manage
#
appgw:
subscriptionId: 260524c9-7a4d-4483-8d85-de54f9c40ae8
resourceGroup: haiche-aks-1-gateway-1
name: appgw1voa5jhws7maoc
usePrivateIP: true
# Setting appgw.shared to "true" will create an AzureIngressProhibitedTarget CRD.
# This prohibits AGIC from applying config for any host/path.
# Use "kubectl get AzureIngressProhibitedTargets" to view and change this.
shared: false
################################################################################
# Specify which Kubernetes namespace the ingress controller will watch
# Default value is "default"
# Leaving this variable out or setting it to blank or empty string would
# result in ingress controller observing all acessible namespaces.
#
kubernetes:
watchNamespace: sample-domain1-ns
################################################################################
# Specify the authentication with Azure Resource Manager
#
# Two authentication methods are available:
# - Option 1: AAD-Pod-Identity (https://github.com/Azure/aad-pod-identity)
# armAuth:
# type: aadPodIdentity
# identityResourceID:
# identityClientID:
armAuth:
type: servicePrincipal
secretJSON: <secret>
################################################################################
# Specify if the cluster is RBAC enabled or not
rbac:
# Specifies whether RBAC resources should be created
create: true
Ingress Controller details
- Output of
kubectl describe pod <ingress controller> . Thepod name can be obtained by running helm list.
Name: ingress-azure-7bb7749d8-q7bm7
Namespace: default
Priority: 0
Node: aks-agentpool-13946896-vmss000001/10.224.0.5
Start Time: Tue, 31 May 2022 18:04:52 +0800
Labels: app=ingress-azure
pod-template-hash=7bb7749d8
release=ingress-azure
Annotations: checksum/config: 54f7501e40746d9d906a2ad5724979802a2e47f15ed8c3bad717d2c9cce9cf5c
prometheus.io/port: 8123
prometheus.io/scrape: true
Status: Running
IP: 10.244.1.12
IPs:
IP: 10.244.1.12
Controlled By: ReplicaSet/ingress-azure-7bb7749d8
Containers:
ingress-azure:
Container ID: containerd://aaa26aa5dd34b3d4b465d99d09eee198b9df107628681f8760ee7e4865e54b74
Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1
Image ID: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:cc131292df265926942e23ca5601a3de66e8feabcb81f705d8f7d84b740f81b6
Port: <none>
Host Port: <none>
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Tue, 31 May 2022 18:21:02 +0800
Finished: Tue, 31 May 2022 18:21:03 +0800
Ready: False
Restart Count: 8
Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
ingress-azure ConfigMap Optional: false
Environment:
AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json
AGIC_POD_NAME: ingress-azure-7bb7749d8-q7bm7 (v1:metadata.name)
AGIC_POD_NAMESPACE: default (v1:metadata.namespace)
AZURE_AUTH_LOCATION: /etc/Azure/Networking-AppGW/auth/armAuth.json
Mounts:
/etc/Azure/Networking-AppGW/auth from networking-appgw-k8s-azure-service-principal-mount (ro)
/etc/appgw/ from azure (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7nb6q (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
azure:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/
HostPathType: Directory
networking-appgw-k8s-azure-service-principal-mount:
Type: Secret (a volume populated by a Secret)
SecretName: networking-appgw-k8s-azure-service-principal
Optional: false
kube-api-access-7nb6q:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 19m default-scheduler Successfully assigned default/ingress-azure-7bb7749d8-q7bm7 to aks-agentpool-13946896-vmss000001
Normal Pulled 19m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1" in 129.696054ms
Normal Pulled 19m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1" in 100.958457ms
Normal Pulled 19m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1" in 75.482394ms
Normal Created 19m (x4 over 19m) kubelet Created container ingress-azure
Normal Started 19m (x4 over 19m) kubelet Started container ingress-azure
Normal Pulling 19m (x4 over 19m) kubelet Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1"
Normal Pulled 19m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1" in 98.417938ms
Warning BackOff 4m54s (x82 over 19m) kubelet Back-off restarting failed container
- Output of `kubectl logs
.
I0531 10:05:47.700084 1 utils.go:114] Using verbosity level 3 from environment variable APPGW_VERBOSITY_LEVEL
I0531 10:05:47.743106 1 supported_apiversion.go:70] server version is: 1.21.9
I0531 10:05:47.763709 1 main.go:118] Using User Agent Suffix='ingress-azure-7bb7749d8-q7bm7' when communicating with ARM
I0531 10:05:47.763995 1 main.go:137] Application Gateway Details: Subscription="260524c9-7a4d-4483-8d85-de54f9c40ae8" Resource Group="haiche-aks-1-gateway-1" Name="appgw1voa5jhws7maoc"
I0531 10:05:47.764060 1 auth.go:38] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json
I0531 10:05:47.765078 1 httpserver.go:57] Starting API Server on :8123
I0531 10:05:48.112538 1 main.go:186] Ingress Controller will observe the following namespaces:sample-domain1-ns
F0531 10:05:48.200347 1 main.go:192] Got a fatal validation error on existing Application Gateway config. Please update Application Gateway or the controller's helm config. Error:Code="ErrorNoPublicIP" Message="Applcation Gateway doesn't have a public IP"
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0x1)
/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1026 +0x8a
k8s.io/klog/v2.(*loggingT).output(0x2858520, 0x3, {0x0, 0x0}, 0xc00022ab60, 0x1, {0x1f73ff3, 0xc00005e800}, 0xc000468360, 0x0)
/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:975 +0x63d
k8s.io/klog/v2.(*loggingT).printDepth(0x0, 0x46056, {0x0, 0x0}, {0x0, 0x0}, 0x13, {0xc000468360, 0x2, 0x2})
/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:735 +0x1ba
k8s.io/klog/v2.(*loggingT).print(...)
/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:717
k8s.io/klog/v2.Fatal(...)
/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1494
main.main()
/azure/cmd/appgw-ingress/main.go:192 +0x176e
goroutine 19 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1169 +0x6a
created by k8s.io/klog/v2.init.0
/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:420 +0xfb
- Any Azure support tickets associated with this issue.
Any update on this? Issue #717 is related to this one. My solution does not allow for AppGW v2 so I'm stuck with v1. It seems moving the SKU validation would be my best bet although I'm not sure if this would be much work. @mscatyao Can you perhaps point me in the right direction?
Same problem here. Have a Standard_v2 application gateway with the EnableApplicationGatewayNetworkIsolation preview, and the controller is throwing
Error:Code="ErrorNoPublicIP" Message="Applcation Gateway doesn't have a public IP"
I'm curious if the AGIC was never updated to handle a fully private appgw?
https://github.com/Azure/application-gateway-kubernetes-ingress/issues/1423#issuecomment-1194455124
UPDATE - I discovered the AKS Addon was installing 1.5.3 while the private appgw was implemented in 1.7.0-RC. Upgrading kubernetes to 1.27.x brings AGIC Addon 1.7.x.
https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/CHANGELOG/CHANGELOG-1.7.md https://github.com/Azure/AKS/blob/master/CHANGELOG.md
