When using Keda as Autoscaler - Deployments Cause All but 1 Pod to Terminate

[matthawley]

Describe the bug We are utilizing Keda as an autoscaler to be based upon current connections. However, when a deployment occurs, all but 1 NGINX Ingress pods are terminated due to replicas: <value> being included within the deployment manifest. This is documented at https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#migrating-deployments-and-statefulsets-to-horizontal-autoscaling

To Reproduce You can follow the steps listed in this blog post: https://www.nginx.com/blog/microservices-march-reduce-kubernetes-latency-with-autoscaling/

Expected behavior When doing a deployment, it should respect the rolling strategy as defined, and not include replicas: <value>.

Your environment

• Version of the Ingress Controller - 3.3.1
• Version of Kubernetes - 1.27
• Kubernetes platform (e.g. Mini-kube or GCP) - Azure Kubernetes

Additional context If there was a way to enable autoscaling without including the built-in hpa

[github-actions[bot]]

Hi @matthawley thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this :slightly_smiling_face:

Cheers!

[haywoodsh]

Hi @matthawley

I'm working on understanding the problem you're facing. Are you encountering this issue while deploying the ingress controller utilizing Helm, and subsequently finding that Helm doesn't generate the manifest required for KEDA?

As a workaround, have you tried deploying the ingress controller directly using the provided manifests? You can find detailed instructions in this documentation: Installation with Manifests | NGINX Ingress Controller

When the autoscaling feature is disabled in Helm (which is the default configuration), the spec:replicas field would be present in the deployment manifest and defaults to 1. This scenario can lead to the termination of the pods. Is that what you observed? https://github.com/nginxinc/kubernetes-ingress/blob/v3.3.1/deployments/helm-chart/templates/controller-deployment.yaml#L13-L15

If this mirrors your scenario, it seems we might need to adjust our Helm chart to ensure it is fully compatible with KEDA.

[matthawley]

Hi @haywoodsh - yes, when we upgrade from helm is when we experience this issue. Helm does generate the right manifests, except that because we have autoscaling disabled (because we're using Keda), that replicas: 1 in the deployment is causing the issue described in the Kubernetes documentation (all but 1 pods are terminated). At this point, our workaround has been to fork the chart and remove the hpa from it so that we can enable "autoscaling" in hopes that the stable chart can support this scenario soon.

[haywoodsh]

Thanks for confirming this @matthawley. I'll bring this back to the wider team to see how we can better support KEDA in our helm chart in future releases.