Nginx pod restarting in a loop after applying configuration changes

[astralko]

What happened:

edited the nginx-ingress-controller configmap: changed allow-snippet-annotations: "false" to

allow-snippet-annotations: "true" annotations-risk-level: Critical use-forwarded-headers: "true"

afterwards my nginx ingress controller pod got errors while reloading the configuration

I0401 08:32:56.115814 1 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"lordabbett", Name:"tenant-websocket-ingress", UID:"a0a86d94-4323-4cd4-86ac-9aefb443dac7", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1222523732", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync E0401 08:32:56.572590 1 controller.go:211] Unexpected failure reloading the backend: exit status 1 2025/04/01 08:32:54 [notice] 549#549: signal process started 2025/04/01 08:32:54 [alert] 549#549: kill(34, 1) failed (3: No such process) nginx: [alert] kill(34, 1) failed (3: No such process) E0401 08:32:56.572655 1 queue.go:131] "requeuing" err=< exit status 1 2025/04/01 08:32:54 [notice] 549#549: signal process started 2025/04/01 08:32:54 [alert] 549#549: kill(34, 1) failed (3: No such process) nginx: [alert] kill(34, 1) failed (3: No such process)

key="tenant/tenant-user-api-service-z9dsc" I0401 08:32:56.572890 1 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"nginx-ingress-controller-8d6857857-ljjf4", UID:"af80e5d9-42cc-4f84-b6c8-b1fc92c723ce", APIVersion:"v1", ResourceVersion:"1222453499", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX: exit status 1 2025/04/01 08:32:54 [notice] 549#549: signal process started 2025/04/01 08:32:54 [alert] 549#549: kill(34, 1) failed (3: No such process) nginx: [alert] kill(34, 1) failed (3: No such process)

it happened over and over, got crashed in a loop. only fixed by scaling the deployment to 0, and then back to 2

What you expected to happen:

I expected the reload of the config to work without crashing. not sure why scaling down and up again fixed the issue

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

---

NGINX Ingress controller Release: 1.12.1 Build: 64780b1 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.25.5

---

Kubernetes version (use kubectl version):

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Debian GNU/Linux 12 (bookworm)
• Kernel (e.g. uname -a): Linux nginx-ingress-controller-79bb8d944d-kr6ld 5.10.233-223.887.amzn2.x86_64 #1 SMP Sat Jan 11 16:55:02 UTC 2025 x86_64 GNU/Linux
• Install tools: helm
• Basic cluster related info:
  • kubectl version: Client Version: v1.28.8-eks-ae9a62a Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.31.6-eks-bc803b4
  • kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-10-33-119.us-east-2.compute.internal Ready 92m v1.31.4-eks-aeac579 10.10.33.119 Amazon Linux 2 5.10.233-223.887.amzn2.x86_64 containerd://1.7.25 ip-10-10-33-125.us-east-2.compute.internal Ready 6d19h v1.31.4-eks-aeac579 10.10.33.125 Amazon Linux 2 5.10.233-223.887.amzn2.x86_64 containerd://1.7.25
• How was the ingress-nginx-controller installed:
  • If helm was used then please show output of helm ls -A | grep -i ingress nginx-ingress-controller kube-system 3 2025-03-30 09:41:32.702978 +0300 IDT deployed nginx-ingress-controller-11.6.12 1.12.1
  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename> USER-SUPPLIED VALUES: defaultBackend: image: registry: xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com repository: gytpol-nginx-ingress-controller tag: 1.27.4-debian-12-r7 global: security: allowInsecureImages: true image: registry: xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com repository: gytpol-nginx-ingress-controller tag: 1.12.1-debian-12-r0

(these image were pulled from docker pull --platform linux/amd64 docker.io/bitnami/nginx-ingress-controller:1.12.1-debian-12-r0 docker pull --platform linux/amd64 docker.io/bitnami/nginx:1.27.4-debian-12-r7 )

• Current State of the controller:
  • kubectl describe ingressclasses Name: alb Labels: app.kubernetes.io/instance=aws-load-balancer-controller app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=aws-load-balancer-controller app.kubernetes.io/version=v2.4.2 helm.sh/chart=aws-load-balancer-controller-1.4.3 Annotations: meta.helm.sh/release-name: aws-load-balancer-controller meta.helm.sh/release-namespace: kube-system Controller: ingress.k8s.aws/alb Events:

Name: nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=nginx-ingress-controller app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=nginx-ingress-controller app.kubernetes.io/version=1.12.1 helm.sh/chart=nginx-ingress-controller-11.6.12 Annotations: meta.helm.sh/release-name: nginx-ingress-controller meta.helm.sh/release-namespace: kube-system Controller: k8s.io/ingress-nginx Events:

• Others:
  • Any other related information like ; we upgraded to this version from a much older one from 2022

Anything else we need to know: