AGIC generated backend pool members for OOMKilled pods of which IP is reused by valid pod

[bbreijer]

Describe the bug AGIC generated backend pool members for OOMKilled pods of which IP is reused by valid pod. The bug surfaced on two separate clusters on the same day.

To Reproduce Steps to reproduce the behavior:

• Make sure you have some OOMKilled pods with an (backend)IP address assigned on domain A in namespace A
• Create new deployments on domain B in namespace B up until the point that one of the new pods gets an IP address of the OOMKilled pod in namespace A
• observe that the IPs are present in the backend pools of both A and B
• observe that traffic of domain A is routed to the backend of domain B (the new, valid pod). Occurrence depends on number of pods in BE pool

Ingress Controller details

• AGIC (ARM deployed): 1.5.2/78cb4021/2022-05-06-08:27T+0000
• Output of `kubectl logs : Available on request or as part of the SR.
• Any Azure support tickets associated with this issue: SR-2207210050001646

[akshaysngupta]

@bbreijer Thanks for reporting this issue. We will investigate what can be done to avoid this issue.

[bbreijer]

Today I heard back from the SR that my issue is the result of a k8s issue: https://github.com/kubernetes/kubernetes/issues/109414 and the fix: https://github.com/kubernetes/kubernetes/pull/110255 .

The issue is fixed in v1.25, which is not yet available for AKS.

[bbreijer]

According to the AKS release calendar, v1.25 is planned (GA) for november 2022: https://docs.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar

[bondido]

Hi, I'm facing kind of simmilar issue on my AKS (kubernetes version 1.23.8). Deployments are configured with horizontalpodautoscalers and it seems that after scaling-in one deployment's replicas count, freed pod's IP can be assigned to newly created pod of another scaled-out replica set without AGIC being notified and as a result not updating backend pool on Application gateway. It seems like a serious issue fo me.

@bbreijer - have you been offered any workaround until v1.25 is goig to be available?

@akshaysngupta - can you confirm bbreijer's info? Do you know when v1.25 is going to be available for AKS as a preview? (according to AKS Kubernetes Release Calendar it should've happened in september)

[bbreijer]

@bbreijer - have you been offered any workaround until v1.25 is goig to be available?

No workaround was offered. We already had logging in place on the KubeEvents, therefore we are notified on OOMKilled pod. At this moment we delete a pod as quickly as possible after storing diagnostic data. Since we continuously optimise the pod settings, OOM's occur not that often. Nevertheless, i think this is a tricky issue which might lead to downtime, from my point of view, our AKS clusters are degraded until the fix is available on AKS GA.

[bondido]

Thaks @bbreijer for the answer. Do You observe the issue only after pods are being OOMKilled? Do You heave any horizontalpodautoscalers on Your AKSes ?

[bbreijer]

Thaks @bbreijer for the answer. Do You observe the issue only after pods are being OOMKilled? Do You heave any horizontalpodautoscalers on Your AKSes ?

We actively observed the issue when pods are OOMKilled and stay (inactive) in the system with that state (i think due to the new one being created on another node, but i'am not sure), the IP is still assigned and the AGIC treats this as a routable IP while k8s reuses the same IP. Result: two entrypoint being routed to the same IP. We don't have horizontalpodautoscalers at this moment.