NGINX: Correctly determine client IP.

[dmeremyanin]

What this PR does / why we need it:

This update ensures that the correct client IP address is determined in Ingress NGINX when the PROXY protocol is enabled. The logic now properly handles proxy chains by checking trusted proxy addresses and using the appropriate forwarded-for headers.

Here's an example when Cloudflare is used:

flowchart LR
    user(User) --> proxy
    proxy(Cloudflare) -- http --> lb
    subgraph VPC
    lb(TCP LB) -- proxy protocol --> nginx(ingress-nginx)
    end

In this scenario, Cloudflare's IP address appears as the client IP. This PR ensures that the actual client IP is correctly identified, even when Cloudflare (or other proxies) are involved.

The algorithm

To ensure the correct client IP is identified, we need to enable use-forwarded-headers and properly configure proxy-real-ip-cidr to trust all intermediate proxies (both within the private network and any external proxies). Caution: By default, the proxy-real-ip-cidr is set to 0.0.0.0/0, which means all IP addresses are trusted.

The process works as follows:

1. Check if the $proxy_protocol_addr is trusted (i.e., whether it's in the proxy-real-ip-cidr list).
2. If trusted, use the configured forwarded-for header (X-Forwarded-For by default).
3. If not trusted, fall back to using the $proxy_protocol_addr.

Once the appropriate header is determined, its value is used with the real_ip_header directive, allowing NGINX to correctly identify the client IP.

---

Issues addressed by this PR:

• https://github.com/kubernetes/ingress-nginx/issues/11623
• https://github.com/kubernetes/ingress-nginx/issues/10749
• https://github.com/kubernetes/ingress-nginx/issues/5233
• https://github.com/kubernetes/ingress-nginx/issues/4731
• and others

In these issues, a workaround was suggested, such as adding the following annotation:

nginx.ingress.kubernetes.io/server-snippet: |
  real_ip_header X-Forwarded-For;

However, this approach may not work correctly in scenarios where clients can connect directly to the TCP load balancer (bypassing the trusted HTTP proxy):

1. Security risk: Any client may be able to forge the X-Forwarded-For header, potentially spoofing the client IP.
2. Incorrect client IP detection: If the X-Forwarded-For header is not passed, the TCP load balancer's IP address will be used as the client IP. In such cases, the proxy_protocol should be used to correctly identify the client IP.

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] CVE Report (Scanner found CVE and adding report)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] Documentation only

It looks more like a new feature because it's now possible to use use-proxy-protocol with use-forwarded-headers configuration option to change the behavior of client IP address identifying.

How Has This Been Tested?

I tested the functionality in a test environment simulating a typical production setup. The environment includes a Layer 7 proxy (Cloudflare) in front of a cloud-based TCP load balancer with the PROXY protocol enabled, and Ingress NGINX Controller within the K8S cluster. To verify the results, I exposed the whoami service using the Ingress Controller and inspected the incoming X-Real-IP and X-Forwarded-For headers reaching the service.

Here's the setup (IPs are fictional):

1. Client IP address is 100.64.0.1
2. HTTP proxy IP address is 198.18.0.1 (passing the real client IP in the X-Forwarded-For header)
3. TCP LB private IP address is 10.1.1.1 (with PROXY protocol enabled)
4. Kubernetes worker node IP address is 10.2.1.1
5. Config option use-proxy-protocol is enabled in Ingress NGINX Controller

Testing Scenarios

With HTTP Proxy (client → HTTP proxy → TCP LB → Ingress Controller)

1. Neither use-forwarded-headers nor proxy-real-ip-cidr are configured.

   Result: 🔴 HTTP proxy IP 198.18.0.1 is identified as the client IP.

2. use-forwarded-headers disabled, proxy-real-ip-cidr set to 198.18.0.0/24 to trust the HTTP proxy subnet.

   Result: 🔴 Kubernetes worker node IP 10.2.1.1 is identified as the client IP.

3. use-forwarded-headers disabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to the trust private network and the HTTP proxy subnet.

   Result: 🔴 HTTP proxy IP 198.18.0.1 is identified as the client IP.

4. use-forwarded-headers enabled, proxy-real-ip-cidr not configured (0.0.0.0/0 by default).

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

5. use-forwarded-headers enabled, proxy-real-ip-cidr not configured, passed custom "X-Forwarded-For: 1.1.1.1" header.

   Result: 🔴 Spoofed IP 1.1.1.1 is used as the client IP.

6. use-forwarded-headers enabled, proxy-real-ip-cidr set to 198.18.0.0/24 to trust the HTTP proxy subnet.

   Result: 🔴 Kubernetes worker node IP 10.2.1.1 is identified as the client IP.

7. use-forwarded-headers enabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to trust the private network and the HTTP proxy subnet.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

8. use-forwarded-headers enabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to trust the private network and the HTTP proxy subnet, passed custom "X-Forwarded-For: 1.1.1.1" header.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

---

Without intermediate HTTP Proxy (client → TCP LB → Ingress Controller)

1. Neither use-forwarded-headers nor proxy-real-ip-cidr are configured.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

2. use-forwarded-headers disabled, proxy-real-ip-cidr set to 198.18.0.0/24 to trust the HTTP proxy subnet.

   Result: 🔴 Kubernetes worker node IP 10.2.1.1 is identified as the client IP.

3. use-forwarded-headers disabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to trust the private network and the HTTP proxy subnet.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

4. use-forwarded-headers enabled, proxy-real-ip-cidr not configured (0.0.0.0/0 by default).

   Result: 🔴 Kubernetes worker node IP 10.2.1.1 is identified as the client IP.

5. use-forwarded-headers enabled, proxy-real-ip-cidr not configured, passed "X-Forwarded-For: 1.1.1.1" header.

   Result: 🔴 Spoofed IP 1.1.1.1 is used as the client IP.

6. use-forwarded-headers enabled, proxy-real-ip-cidr set to 198.18.0.0/24 to trust the HTTP proxy subnet.

   Result: 🔴 Kubernetes worker node IP 10.2.1.1 is identified as the client IP.

7. use-forwarded-headers enabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to trust the private network and the HTTP proxy subnet.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

8. use-forwarded-headers enabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to trust the private network and the HTTP proxy subnet, passed custom "X-Forwarded-For: 1.1.1.1" header.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

Conclusion and Questions

1. The default client IP identification behavior remains unchanged. The identification process only behaves differently when both use-proxy-protocol and use-forwarded-headers are enabled.
2. I've added only two end-to-end tests to cover the main functionality because they are very slow to run, so it's a bit of a compromise.
3. Wouldn't it be better to set proxy-real-ip-cidr to something like 10.0.0.0/8 by default? Setting this default would help prevent misconfigurations and ensure that only trusted internal IP ranges are accepted, reducing the risk of spoofed IPs.

Checklist:

• [x] My change requires a change to the documentation.
• [x] I have updated the documentation accordingly.
• [x] I've read the CONTRIBUTION guide
• [x] I have added unit and/or e2e tests to cover my changes.
• [x] All new and existing tests passed.