ingress-nginx
ingress-nginx copied to clipboard
kubernetes
Add a way to customize X-Forwarded-For
Hello everyone,
I am trying to configure my ingress-nginx controller so that the applications hosted in a Kubernetes cluster can see the real remote IP of the client.
My case is only based on X-Forwarded-For header and I cannot move the PROXY
protocol immediately.
Let's image the situation of application hosted in AWS that can be accessed in different ways:
- a client can send a query to CloudFront which forwards to an Application Load Balancer
- a client can sent a query directly to an Application Load Balancer
This means the traversed trusted (inside AWS) proxies are either in the VPC (second case) our outside of the VPC (first case).
More visually, let's take two clients:
+----------+ +----------+
| Client 1 | | Client 2 |
+----------+ +----------+
| |
| |
+---------+---------------AWS----------------+----------------------+
| | | |
| V | |
| +-------------+ | |
| | CloudFront | | |
| +-------------+ | |
| | | |
| | | |
| +-------+--My VPC--------------------------+------------+ |
| | | | | |
| | | | | |
| | +--------+ | | |
| | | API-GW | | | |
| | +--------+ | | |
| | | | | |
| | +-------------------------------+ | | |
| | | | | |
| | | | | |
| | V V | |
| | +-----------EKS (k8s)----------+ +---------+ | |
| | | | | EC2 ELB | | |
| | | | +---------+ | |
| | | +---------------+ | | | |
| | | | Nginx Ingress |<----+---------+ | |
| | | +---------------+ | | |
| | | | | | |
| | | V | | |
| | | +-------------+ | | |
| | | | Application | | | |
| | | | Pods | | | |
| | | +-------------+ | | |
| | | | | |
| | +------------------------------+ | |
| +-------------------------------------------------------+ |
+-------------------------------------------------------------------+
With the following configuration, as described in the documentation (https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-real-ip-cidr):
use-forwarded-headers: "true"
proxy-real-ip-cidr: "${VPC_RANGE}"
The applications will receive the correct Client 2 IP address (because the client is right at the boundary of the VPC). But the application will only see the CloudFront machine IP when the requests are sent by Client 1.
One solution to cope with this issue could be to add the public IPs of CloudFront that are published by AWS at https://ip-ranges.amazonaws.com/ip-ranges.json. But that introduce some static configuration and regular updates of the AWS IPs.
But a more dynamic solution could be to rely upon the CloudFront headers. For
Client 1, I think the best source of true should be in the added header
CloudFront-Viewer-Address.
So I was thinking to a solution in which I could create a
$custom_forwarded_for variable which would be used to set the correct value
to the X-Forwarded-For header to send to the application.
Something like this could do the trick:
map $http_CloudFront_Viewer_Address $custom_forwarded_for {
"~*^(?<cf_viewer_address>.*):\d{1,5}$" $cf_viewer_address;
default remote_addr;
}
The problem is that it cannot be used easily. The default template force this line:
proxy_set_header X-Forwarded-For $remote_addr;
This line is as far as I know impossible to override. Using
more_set_input_headers will not help and reusing proxy_set_header would
only append the list.
So at that point I can see only solution:
I was wondering whether there was a more straightforward way to send the custom
X-Forwarded-For header.
I was thinking of an new configuration like custom-forwarded-for-variable. For
instance, we could have the following configuration in the ConfigMap:
http-snippet: |
map $http_CloudFront_Viewer_Address $custom_forwarded_for {
"~*^(?<cf_viewer_address>.*):\d{1,5}$" $cf_viewer_address;
default remote_addr;
}
custom_forwarded-for-variable: "custom_forwarded_for"
The official template could then be modified that way:
{{ if and $all.Cfg.UseForwardedHeaders $all.Cfg.ComputeFullForwardedFor }}
{{ $proxySetHeader }} X-Forwarded-For $full_x_forwarded_for;
{{ else if $all.Cfg.CustomForwardedForVariable }}
{{ $proxySetHeader }} X-Forwarded-For ${{ $all.Cfg.CustomForwardedForVariable }};
{{ else }}
{{ $proxySetHeader }} X-Forwarded-For $remote_addr;
{{ end }}
There are probably better solution that I would be happy to read.
I can see there are several issues related to mine (for instance), and think with this change, it would simply the management of other connectivities (like taking the CloudFlare's CF-Remote-IP header when needed).
Thanks in advance for your feedbacks or better solution to solve this usecase.
This issue is currently awaiting triage.
If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
@stac47 There is another load balancer when you deploy it into AWS that would need the settings in the ELB like the other one. Ingress deploys a service type of LoadBalancer and the cloud controller deploys one for Ingress to use.
@strongjz I have a very similar case. I want to support both NLB using (proxy protocol) and ALB using client port preservation, but use-forwarded-headers behavior is odd ie. it removes the client port. We often need this for government/police requests. Potentially in the future I will also need to support CloudFront 😅
So far, I have had to use two separate nginx ingress for ALB and NLB (it depends on use case)
For ALB
proxySetHeaders:
X-Forwarded-Port: '$remote_port'
X-Forwarded-For: '$remote_addr:$remote_port'
For NLB
proxySetHeaders:
X-Forwarded-Port: '$proxy_protocol_port'
X-Forwarded-For: '$proxy_protocol_addr:$proxy_protocol_port'
For .NET we need the addr:port because how .NET parses forwarded for header.
With proxy protocol the x-forwarded-for gets an ip at least without the use-forwarded-headers=false and use-proxy-protocol=true
with ALB and client port preservation the X-Original-Forwarded-For is fine and has a port but than X-Forwarded-For seems to remove the port with use-forwarded-headers=true
So that's why I added the proxySetHeaders
So yes it seems very much like I want some sort of customization that allows me for one to have conditional so I could use the same load balancer with minimal configuration change.
Example of the headers:
"X-Forwarded-For": [
"10.10.20.139",
"10.10.20.139:22737"
],
"X-Forwarded-Host": [
"host.kube.my"
],
"X-Forwarded-Port": [
"443",
"22737"
],
"X-Forwarded-Proto": [
"https"
],
"X-Forwarded-Scheme": [
"https"
],
"X-Forwarded-For": [
"99.99.99.123",
"99.99.99.123:24800"
],
"X-Forwarded-Host": [
"host.kube.my"
],
"X-Forwarded-Port": [
"443",
"24800"
],
"X-Forwarded-Proto": [
"https"
],
"X-Forwarded-Scheme": [
"https"
],
@strongjz Hello, I updated the schema to be closer to my infrastructure.
There is another load balancer when you deploy it into AWS that would need the settings in the ELB like the other one. Ingress deploys a service type of LoadBalancer and the cloud controller deploys one for Ingress to use.
Not sure to understand. Do you mean my proposal would not work?
We have slightly the same issue, but with CloudFlare, not CloudFront.
Temporary solution is splitting Ingresses and add annotation to the second one (for clients with CF):
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header X-Forwarded-For $http_cf_connecting_ip;
Similar issues:
- https://github.com/kubernetes/ingress-nginx/issues/7761
- https://github.com/kubernetes/ingress-nginx/issues/4731
- https://github.com/kubernetes/ingress-nginx/issues/3738
@dmeremyanin I not entirely sure how #12768 fixes customizing X-Forwarded-For?
I cannot get it to include the port number and even with proxy-protocol.
Even when I use forwarded-for-header: "CloudFront-Viewer-Address" it removes the port number.
The actual result that gets me the port number:
proxySetHeaders:
X-Forwarded-For: '$http_cloudfront_viewer_address'
X-Forwarded-Proto: 'https'
config:
proxy-real-ip-cidr: 0.0.0.0/0
use-forwarded-headers: "true"
use-proxy-protocol: "true"
forwarded-for-header: "CloudFront-Viewer-Address"
This ensure all forwarded headers are mostly correct and because how .NET parses x-forwarded-for I can actually get the port number.
But the headers look like this coming into .NET, the remotePort, remoteIp and scheme is how .NET using the UseForwardedHeaders behavior parses the headers below
{
"remotePort": 60743,
"remoteIp": "1.2.3.4",
"host": "xforwarded.example.app",
"scheme": "https",
"headers": {
"Accept": [
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
],
"Host": [
"xforwarded.example.app"
],
"User-Agent": [
"Mozilla/5.0 (X11; Linux x86_64; rv:141.0) Gecko/20100101 Firefox/141.0"
],
"Accept-Encoding": [
"br,gzip"
],
"Accept-Language": [
"en-US,en;q=0.5"
],
"traceparent": [
"00-b450335a350d76f42cd6dc37bc864148-64fc24fecf6f35a4-00"
],
"Upgrade-Insecure-Requests": [
"1"
],
"Via": [
"2.0 f14d816589c938c13b4401641d90dcd2.cloudfront.net (CloudFront)"
],
"X-Request-ID": [
"de92315e1a1dab8b39ee3945c3ca946c"
],
"X-Real-IP": [
"1.2.3.4"
],
"X-Forwarded-For": [
"1.2.3.4",
"1.2.3.4:60743"
],
"X-Forwarded-Host": [
"xforwarded.example.app"
],
"X-Forwarded-Port": [
"80"
],
"X-Forwarded-Proto": [
"http",
"https"
],
"X-Forwarded-Scheme": [
"http"
],
"X-Scheme": [
"http"
],
"X-Original-Forwarded-For": [
"1.2.3.4:60743"
],
"CloudFront-Viewer-Address": [
"1.2.3.4:60743"
],
"X-Amz-Cf-Id": [
"snip"
],
"CloudFront-Viewer-ASN": [
"1111"
],
"CloudFront-Viewer-TLS": [
"TLSv1.3:TLS_AES_128_GCM_SHA256:connectionReused"
],
"sec-gpc": [
"1"
],
"sec-fetch-dest": [
"document"
],
"sec-fetch-mode": [
"navigate"
],
"sec-fetch-site": [
"none"
],
"sec-fetch-user": [
"?1"
],
"priority": [
"u=0, i"
],
"CloudFront-Is-Mobile-Viewer": [
"false"
],
"CloudFront-Is-Tablet-Viewer": [
"false"
],
"CloudFront-Is-SmartTV-Viewer": [
"false"
],
"CloudFront-Is-Desktop-Viewer": [
"true"
],
"CloudFront-Is-IOS-Viewer": [
"false"
],
"CloudFront-Is-Android-Viewer": [
"false"
],
"CloudFront-Viewer-HTTP-Version": [
"2.0"
],
"CloudFront-Viewer-Country": [
"IE"
],
"CloudFront-Viewer-City": [
"Dublin"
],
"CloudFront-Forwarded-Proto": [
"https"
],
"X-Forwarded-For-Proxy-Protocol": [
"1.2.3.4:60743"
]
}
}
For .NET I use the builtin functionality to parse XForwardedFor headers.
services.Configure<ForwardedHeadersOptions>(
options =>
{
options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
options.KnownNetworks.Clear();
options.KnownProxies.Clear();
});
app.UseForwardedHeaders();
@jetersen just seeing this, sorry for the delay.
If I understand correctly, you're trying to get the original source port (e.g., from CloudFront-Viewer-Address) included in X-Forwarded-For, but it's getting stripped?
PR #12768 doesn't handle ports – it only adds the X-Forwarded-For-Proxy-Protocol header (which you're seeing in your .NET app) and configures NGINX to use it as the source IP via the real_ip_header directive.
Here's the NGINX docs for that directive: https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
That directive parses the header and sets two variables: $remote_addr and $remote_port. $remote_addr is just the IP, not IP + port. Then in the NGINX template, $remote_addr is used to populate the X-Real-IP and X-Forwarded-For headers: https://github.com/kubernetes/ingress-nginx/blob/f598b64ade33367c43eec8f2a1debab2060fe69f/rootfs/etc/nginx/template/nginx.tmpl#L1063
I think that's why you're seeing just an IP address in the header.
The issue is about customizing X-Forwarded-For which I think would include getting the remote port. I have created a commit but have had trouble figuring out how to build an ingress image to test it out. https://github.com/jetersen/ingress-nginx/commit/5b0d4fece090a363c261498887b8c66899b116ba