ingress-nginx icon indicating copy to clipboard operation
ingress-nginx copied to clipboard

security: proxy-real-ip-cidr does NOT control use-forwarded-headers despite what the doc says

Open Hexcles opened this issue 3 years ago • 12 comments

What happened:

  • Set use-forwarded-headers to true and proxy-real-ip-cidr to a trusted L7 proxy
  • Send a request with X-Forwarded-Host and/or X-Forwarded-Proto directly to the ingress without going through the L7 proxy

What you expected to happen:

X-Forwarded-* should be discarded by ingress according to documentation:

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-real-ip-cidr

If use-forwarded-headers or use-proxy-protocol is enabled, proxy-real-ip-cidr defines the default IP/network address of your external load balancer.

which suggests (at least to me) that X-Forwarded-* would only be kept if the requests are coming from the trusted proxy-real-ip-cidr, but in reality they don't seem to interact at all -- they are unconditionally preserved as long as use-forwarded-headers is true:

https://github.com/kubernetes/ingress-nginx/blob/739e85ce7cf2dbc13b710036a5c3fc465425c6e2/rootfs/etc/nginx/lua/lua_ingress.lua#L117-L131

This essentially allows any caller to spoof the hostname regardless of proxy-real-ip-cidr.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): v1.2.1

Kubernetes version (use kubectl version): 1.20

Environment: (unrelated) Anything else we need to know:

This is either a security bug or a documentation error.

Hexcles avatar Oct 14 '22 19:10 Hexcles

@Hexcles: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Oct 14 '22 19:10 k8s-ci-robot

can you write step by step instructions that can be used to reproduce the problem in a minikube or a kind cluster.

longwuyuan avatar Oct 15 '22 08:10 longwuyuan

Sure, in a fresh minikube, install and configure ingress-nginx as follows:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml

echo "
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: foo-bar
    annotations:
      kubernetes.io/ingress.class: nginx
  spec:
    rules:
    - host: foo.bar
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: http-svc
              port: 
                number: 80
---
  apiVersion: v1
  data:
    allow-snippet-annotations: \"true\"
    # These are KEY!
    use-forwarded-headers: \"true\"
    proxy-real-ip-cidr: 8.8.8.8  # Just a random IP that requests will never originate from.
  kind: ConfigMap
  metadata:
    labels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
      app.kubernetes.io/version: 1.4.0
    name: ingress-nginx-controller
    namespace: ingress-nginx
" | kubectl apply -f -

Now make a request:

kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H 'Host: foo.bar' -H 'x-forwarded-proto: https' -H 'x-forwarded-host: malicious' localhost

which gives you

Pod Information:
        node name:      minikube
        pod name:       http-svc-6f46856544-mx8sg
        pod namespace:  default
        pod IP: 172.17.0.3

Server values:
        server_version=nginx: 1.14.2 - lua: 10015

Request Information:
        client_address=172.17.0.5
        method=GET
        real path=/
        query=
        request_version=1.1
        request_scheme=http
        request_uri=http://malicious:8080/

Request Headers:
        accept=*/*
        host=malicious
        user-agent=curl/7.83.1
        x-forwarded-for=127.0.0.1
        x-forwarded-host=malicious
        x-forwarded-port=80
        x-forwarded-proto=https
        x-forwarded-scheme=https
        x-real-ip=127.0.0.1
        x-request-id=5adc6326ff78e047bd4a251f58a4432e
        x-scheme=https

Request Body:
        -no body in request-

Notice how x-forwarded-* and even host got spoofed by the caller even though the caller is from localhost, not 8.8.8.8.

Recommended resolution:

  • either fix the documentation to clarify that proxy-real-ip-cidr only guards x-forwarded-for (or any header used by enable-real-ip), but not the rest of x-forwarded-* headers (use-forwarded-headers)
  • or fix the Lua code linked in the original description to only trust x-forwarded-* headers from the allowlisted IPs.

Hexcles avatar Oct 15 '22 14:10 Hexcles

THis reproduce produce procedure needs to be simpler and make it possible for anyone to copy/paste on their minikube or kind cluster.

This reproduce procedure aassumes a bunch of things so it may cause problems.

Can you help and write a procedure that someone can just copy/paste in their minikube or kind cluster, and reproduce even without knowing much about the features.

longwuyuan avatar Oct 16 '22 09:10 longwuyuan

/remove-kind bug Once the data is available on the details of the bug, we can change the label

longwuyuan avatar Oct 16 '22 09:10 longwuyuan

@longwuyuan I'm sorry. Could you point out which part of the reproduction steps are not clear? I didn't assume anything and reproduced the issue in a bare minikube using pretty much the same procedure as the commented out steps in the issue template. I even highlighted the important customization (the ConfigMap) that leads to the issue using a comment.

Hexcles avatar Oct 16 '22 13:10 Hexcles

Oh is it because of the $POD_NAME env var not found? That's actually a bug in your issue template: https://github.com/kubernetes/ingress-nginx/blob/739e85ce7cf2dbc13b710036a5c3fc465425c6e2/.github/ISSUE_TEMPLATE/bug_report.md?plain=1#L130-L131 the first line would actually produce a few pod names and hence the second line doesn't work.

Let me give you a better reproduction step:

POD_NAME=$(k get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o NAME)
kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H 'Host: foo.bar' localhost

Hexcles avatar Oct 16 '22 13:10 Hexcles

@longwuyuan The steps are pretty clear to me, and I was able to reproduce the issue easily.

komljen avatar Oct 17 '22 14:10 komljen

# setup
kind create cluster
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml
kubectl wait -n ingress-nginx deployment ingress-nginx-controller --for=condition=Available=True --timeout=90s
POD_NAME=$(kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o NAME)

# config
echo "
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: foo-bar
    annotations:
      kubernetes.io/ingress.class: nginx
  spec:
    rules:
    - host: foo.bar
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: http-svc
              port: 
                number: 80
---
  apiVersion: v1
  data:
    allow-snippet-annotations: \"true\"
    # These are KEY!
    use-forwarded-headers: \"true\"
    proxy-real-ip-cidr: 8.8.8.8  # Just a random IP that requests will never originate from.
  kind: ConfigMap
  metadata:
    labels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
      app.kubernetes.io/version: 1.4.0
    name: ingress-nginx-controller
    namespace: ingress-nginx
" | kubectl apply -f -

# run
kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H 'Host: foo.bar' -H 'x-forwarded-proto: https' -H 'x-forwarded-host: malicious' localhost

crinjes avatar Oct 31 '22 17:10 crinjes

We just traced back a security issue in our cluster to this root cause. I'm undecided what should shock us more? That an obvious security oversight like this even happened in the heart of kubernetes, or how this issue was treated the 3 years since it has been reported.

podtom avatar Sep 05 '25 10:09 podtom

Is there any update on a bug fix? This is a critical security vulnerability

surya-harvey avatar Dec 02 '25 05:12 surya-harvey

In case you missed the news, this project is being retired https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/


Sent from my cellphone.

On Mon, Dec 1, 2025, 21:14 Surya Keswani @.***> wrote:

surya-harvey left a comment (kubernetes/ingress-nginx#9163) https://github.com/kubernetes/ingress-nginx/issues/9163#issuecomment-3600226413

Is there any update on a bug fix? This is a critical security vulnerability

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/9163#issuecomment-3600226413, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6ZDESA33LUZGUNXLYYE337UN5FAVCNFSM6AAAAACFWX47TWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMMBQGIZDMNBRGM . You are receiving this because you were mentioned.Message ID: @.***>

Hexcles avatar Dec 02 '25 05:12 Hexcles