ingress-nginx
ingress-nginx copied to clipboard
kubernetes
Ingresses Flapping between Private IPs and Cluster IP
What happened:
All my ingress(es) in our kubernetes rke2 cluster flap using the nodes internal IPs to the Cluster IPs. This behavior breaks any ingress nginx rewrites that take place at the nginx controller layer.
What you expected to happen:
Ingress NGINX rewrites are routed properly to the right ingress.
NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):
bash-5.1$ /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.2.0
Build: a2514768cd282c41f39ab06bda17efefc4bd233a
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.10
-------------------------------------------------------------------------------
Kubernetes version:
🐈 [bel5290@L5-LST-BEL5290M ~] kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.31.9+rke2r1
Environment: Development
-
Cloud provider or hardware configuration:
-
OS (e.g. from /etc/os-release):
-
Kernel: Linux ulkdevctl1.ad.psu.edu 5.14.0-570.18.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 27 21:47:45 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux
-
Install tools:
- Rancher
- RKE2
-
Basic cluster related info:
-
kubectl versionClient Version: v1.31.1 Kustomize Version: v5.4.2 Server Version: v1.31.9+rke2r1 -
kubectl get nodes -o wideNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ulkdevbat1 Ready,SchedulingDisabled worker 13d v1.31.9+rke2r1 192.168.20.12AlmaLinux 9.6 (Sage Margay) 5.14.0-570.18.1.el9_6.x86_64 containerd://2.0.5-k3s1 ulkdevbat2 Ready,SchedulingDisabled worker 13d v1.31.9+rke2r1 192.168.20.13 AlmaLinux 9.6 (Sage Margay) 5.14.0-570.18.1.el9_6.x86_64 containerd://2.0.5-k3s1 ulkdevctl2 Ready control-plane,etcd,master 13d v1.31.9+rke2r1 192.168.20.2 AlmaLinux 9.6 (Sage Margay) 5.14.0-570.17.1.el9_6.x86_64 containerd://2.0.5-k3s1 ulkdevctl3 Ready control-plane,etcd,master 7d4h v1.31.9+rke2r1 192.168.20.3 AlmaLinux 9.6 (Sage Margay) 5.14.0-570.18.1.el9_6.x86_64 containerd://2.0.5-k3s1 ulkdevwrk2 Ready worker 13d v1.31.9+rke2r1 192.168.20.5 AlmaLinux 9.6 (Sage Margay) 5.14.0-570.17.1.el9_6.x86_64 containerd://2.0.5-k3s1 ulkdevwrk3 Ready,SchedulingDisabled worker 13d v1.31.9+rke2r1 192.168.20.6 AlmaLinux 9.6 (Sage Margay) 5.14.0-570.17.1.el9_6.x86_64 containerd://2.0.5-k3s1 ulkdevwrk4 Ready worker 13d v1.31.9+rke2r1 192.168.20.7 AlmaLinux 9.6 (Sage Margay) 5.14.0-570.17.1.el9_6.x86_64 containerd://2.0.5-k3s1 ulkdevwrk5 Ready worker 7d4h v1.31.9+rke2r1 192.168.20.8 AlmaLinux 9.6 (Sage Margay) 5.14.0-570.18.1.el9_6.x86_64 containerd://2.0.5-k3s1 ulkdevwrk6 Ready,SchedulingDisabled worker 13d v1.31.9+rke2r1 192.168.20.9 AlmaLinux 9.6 (Sage Margay) 5.14.0-570.17.1.el9_6.x86_64 containerd://2.0.5-k3s1 ulkdevwrk7 Ready,SchedulingDisabled worker 13d v1.31.9+rke2r1 192.168.20.10 AlmaLinux 9.6 (Sage Margay) 5.14.0-570.17.1.el9_6.x86_64 containerd://2.0.5-k3s1 ulkdevwrk8 Ready,SchedulingDisabled worker 13d v1.31.9+rke2r1 192.168.20.11 AlmaLinux 9.6 (Sage Margay) 5.14.0-570.17.1.el9_6.x86_64 containerd://2.0.5-k3s1
-
-
How was the ingress-nginx-controller installed:
- Installed using argocd and helm:
- Version:
source:
repoURL: https://kubernetes.github.io/ingress-nginx
targetRevision: 4.11.3
chart: ingress-nginx
- Current State of the controller:
kubectl describe ingressclasses🐈 [bel5290@L5-LST-BEL5290M ~] kubectl describe ingressclasses Name: nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.2.0 helm.sh/chart=ingress-nginx-4.1.0 Annotations: argocd.argoproj.io/tracking-id: ingress-nginx:networking.k8s.io/IngressClass:ingress-nginx/nginx ingressclass.kubernetes.io/is-default-class: true Controller: k8s.io/ingress-nginx Events:
🐈 [bel5290@L5-LST-BEL5290M ~] kubectl -n ingress-nginx describe pod ingress-nginx-controller-9fb6c4fdb-mvrsr
Name: ingress-nginx-controller-9fb6c4fdb-mvrsr
Namespace: ingress-nginx
Priority: 0
Service Account: ingress-nginx
Node: ulkdevwrk2/192.168.20.5
Start Time: Tue, 17 Jun 2025 08:27:42 -0400
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=9fb6c4fdb
Annotations: cni.projectcalico.org/containerID: ee23b5a7f8353b7620098003cc3ec6330b434c056ea4b098b6fd78fd6c9b5511
cni.projectcalico.org/podIP: 10.42.9.24/32
cni.projectcalico.org/podIPs: 10.42.9.24/32
Status: Running
IP: 10.42.9.24
IPs:
IP: 10.42.9.24
Controlled By: ReplicaSet/ingress-nginx-controller-9fb6c4fdb
Containers:
controller:
Container ID: containerd://74ddb25787d596331895a1fba32b1203d092c44c1184c87947180089d43feac6
Image: k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185
Image ID: k8s.gcr.io/ingress-nginx/controller@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185
Ports: 80/TCP, 443/TCP, 10254/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
--election-id=ingress-controller-leader
--controller-class=k8s.io/ingress-nginx
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--watch-ingress-without-class=true
--default-ssl-certificate=ingress-nginx/star-uldev-k8s
State: Running
Started: Tue, 17 Jun 2025 08:27:56 -0400
Ready: True
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-9fb6c4fdb-mvrsr (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vcfh5 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-vcfh5:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
🐈 [bel5290@L5-LST-BEL5290M ~] kubectl -n ingress-nginx describe svc ingress-nginx-controller
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.2.0
helm.sh/chart=ingress-nginx-4.1.0
Annotations: argocd.argoproj.io/tracking-id: ingress-nginx:/Service:ingress-nginx/ingress-nginx-controller
field.cattle.io/publicEndpoints:
[{"addresses":["128.118.122.72"],"port":30080,"protocol":"TCP","serviceName":"ingress-nginx:ingress-nginx-controller","allNodes":true},{"a...
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.249.104
IPs: 10.43.249.104
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 30080/TCP
Endpoints: 10.42.9.28:80,10.42.9.24:80,10.42.9.53:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 30443/TCP
Endpoints: 10.42.9.28:443,10.42.9.24:443,10.42.9.53:443
Session Affinity: None
External Traffic Policy: Local
Internal Traffic Policy: Cluster
Events: <none>
- Others:
- Any other related information like ;
- copy/paste of the snippet (if applicable)
kubectl describe ...of any custom configmap(s) created and in use- Any other related information that may help
- Any other related information like ;
How to reproduce this issue:
When I run a kubectl get ingress -A -o wide the ingresses return the private IPs as part of the ADDRESS:
🐈 [bel5290@L5-LST-BEL5290M spells] kubectl get ingress -A -o wide
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
argo argo-workflows-server nginx argo.uldev.k8s.libraries.psu.edu 192.168.20.10,192.168.20.11,192.168.20.12,192.168.20.13,192.168.20.5,192.168.20.6,192.168.20.7,192.168.20.8,192.168.20.9 80 8d
argocd argocd-appset-ingress nginx argocd-appset.uldev.k8s.libraries.psu.edu 192.168.20.10,192.168.20.11,192.168.20.12,192.168.20.13,192.168.20.5,192.168.20.6,192.168.20.7,192.168.20.8,192.168.20.9 80, 443 12d
argocd argocd-server-ingress nginx argocd.uldev.k8s.libraries.psu.edu 192.168.20.10,192.168.20.11,192.168.20.12,192.168.20.13,192.168.20.5,192.168.20.6,192.168.20.7,192.168.20.8,192.168.20.9 80, 443 12d
harbor harbor-harbor-ingress
But these will flap and turn to the ingress nginx cluster ip:
🐈 [bel5290@L5-LST-BEL5290M spells] kubectl get ingress -A -o wide
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
argo argo-workflows-server nginx argo.uldev.k8s.libraries.psu.edu 10.43.249.104 80 8d
argocd argocd-appset-ingress nginx argocd-appset.uldev.k8s.libraries.psu.edu 10.43.249.104 80, 443 12d
argocd argocd-server-ingress nginx argocd.uldev.k8s.libraries.psu.edu 10.43.249.104 80, 443 12d
harbor harbor-harbor-ingress
