The AGIC pod become not running

[JiayangZhou]

Describe the bug I have two AGIC pods deployed in a cluster and one of them is running fine (updating the appgw normally), but the other one is not in READY state (not updating the appgw), see the logs below and any ideas?

The AGIC is installed via helm, the specific version is application-gateway-kubernetes-ingress-staging/ingress-azure build number 32189.

To Reproduce Steps to reproduce the behavior:

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.

I0422 12:24:25.884393       1 reflector.go:255] Listing and watching *v1beta1.Ingress from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167
E0422 12:24:25.897591       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
I0422 12:25:19.190168       1 reflector.go:255] Listing and watching *v1beta1.Ingress from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167
E0422 12:25:19.197227       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
I0422 12:26:11.909281       1 reflector.go:255] Listing and watching *v1beta1.Ingress from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167
E0422 12:26:11.919917       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
I0422 12:27:00.787450       1 reflector.go:255] Listing and watching *v1beta1.Ingress from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167
E0422 12:27:00.794544       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
I0422 12:28:00.491178       1 reflector.go:255] Listing and watching *v1beta1.Ingress from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167
E0422 12:28:00.499707       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)

• Output of `kubectl logs .

PS C:\Users\jzhou1> kubectl describe ingress grid-system-private-application-lb-integration-ingress -n m3dev02-env
Name:             grid-system-private-application-lb-integration-ingress
Namespace:        m3dev02-env
Address:          100.112.50.199
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                                                   Path  Backends
  ----                                                   ----  --------
  m3dev02-integration.m3.azdevlocal.inforcloudsuite.com
                                                         /   grid-system-private-application-lb:37443 ()
Annotations:                                             appgw.ingress.kubernetes.io/appgw-ssl-certificate: m3dev02-integration-m3-azdevlocal-inforcloudsuite-com
                                                         appgw.ingress.kubernetes.io/appgw-trusted-root-certificate: m3dev02_cert
                                                         appgw.ingress.kubernetes.io/backend-hostname: m3dev02-integration.m3.azdevlocal.inforcloudsuite.com
                                                         appgw.ingress.kubernetes.io/backend-protocol: https
                                                         appgw.ingress.kubernetes.io/health-probe-hostname: m3dev02-integration.m3.azdevlocal.inforcloudsuite.com
                                                         appgw.ingress.kubernetes.io/health-probe-path: /grid/healthCheck
                                                         appgw.ingress.kubernetes.io/override-frontend-port: 443
                                                         appgw.ingress.kubernetes.io/request-timeout: 600
                                                         appgw.ingress.kubernetes.io/rewrite-rule-set: oauth_rewrite_ruleset
                                                         appgw.ingress.kubernetes.io/use-private-ip: true
                                                         kubernetes.io/ingress.class: application-gateway-private
                                                         meta.helm.sh/release-name: application-gateway-ingress-private
                                                         meta.helm.sh/release-namespace: m3dev02-env
Events:                                                  <none>
PS C:\Users\jzhou1> kubectl describe ingress grid-system-public-app-lb-saml-ingress -n m3dev02-env
Name:             grid-system-public-app-lb-saml-ingress
Namespace:        m3dev02-env
Address:          20.80.208.232
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                                  Path  Backends
  ----                                  ----  --------
  m3dev02.m3.azdev.inforcloudsuite.com
                                        /   grid-system-public-app-lb:33443 ()
Annotations:                            appgw.ingress.kubernetes.io/appgw-ssl-certificate: m3dev02-m3-azdev-inforcloudsuite-com
                                        appgw.ingress.kubernetes.io/appgw-trusted-root-certificate: m3dev02_cert
                                        appgw.ingress.kubernetes.io/backend-hostname: m3dev02.m3.azdev.inforcloudsuite.com
                                        appgw.ingress.kubernetes.io/backend-protocol: https
                                        appgw.ingress.kubernetes.io/health-probe-hostname: m3dev02.m3.azdev.inforcloudsuite.com
                                        appgw.ingress.kubernetes.io/health-probe-path: /grid/healthCheck
                                        appgw.ingress.kubernetes.io/override-frontend-port: 443
                                        appgw.ingress.kubernetes.io/request-timeout: 600
                                        kubernetes.io/ingress.class: application-gateway-public
                                        meta.helm.sh/release-name: application-gateway-ingress-public
                                        meta.helm.sh/release-namespace: m3dev02-env
Events:                                 <none>

• Any Azure support tickets associated with this issue.

[JiayangZhou]

The issue is resolved though by deleting the Not Ready AGIC pod, it became ready but how could I look into the cause to prevent it from happening in the future.

[severity1]

I am experiencing a variation of this issue. This is happening for me on kubernetes v1.23.3

I0428 00:25:37.765314       1 auth.go:46] Creating authorizer from Azure Managed Service Identity
I0428 00:25:37.765335       1 httpserver.go:57] Starting API Server on :8123
I0428 00:25:48.492748       1 main.go:183] Ingress Controller will observe all namespaces.
I0428 00:25:48.613142       1 context.go:138] k8s context run started
I0428 00:25:48.613168       1 context.go:188] Waiting for initial cache sync
E0428 00:25:48.615665       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:25:48.617230       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:25:51.011040       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:25:55.937539       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:26:06.970103       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:26:31.776209       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:27:09.574870       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:28:04.555823       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:28:35.976707       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:29:14.361793       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)