ingress-nginx
ingress-nginx copied to clipboard
kubernetes
Proxied Host and X-Forwarded-Host headers can contain arbitrary values when request-line contains full URI
NGINX Ingress controller version: dev for 0.32 because of #5457, 0.34.1
Kubernetes version (use kubectl version): 1.18.2
Environment:
- Cloud provider or hardware configuration: Bare metal
- OS (e.g. from /etc/os-release): Ubuntu 18.04.4 LTS
- Kernel (e.g.
uname -a): 4.15.0-88-generic - Install tools: Kind
- Others:
What happened:
Usually the Host and X-Forwarded-Host of the proxied request match the host field of the matching ingressRule of the ingress resource. Given an ingress resource for host 'foo.bar' pointing to an echoserver instance this looks like:
$ kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H 'Host: foo.bar' localhost | grep host=
host=foo.bar
x-forwarded-host=foo.bar
But for requests that contain a full URI in the request-line of the HTTP request the proxied X-Forwarded-Host and Host headers can contain arbitrary values:
$ kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H "Host: can-be-anything" -x http://127.0.0.1:80/ http://foo.bar/ | grep host=
host=can-be-anything
x-forwarded-host=can-be-anything
The hostname that was used to match the request is also not present in any other request header:
$ kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H "Host: can-be-anything" -x http://127.0.0.1:80/ http://foo.bar/ | grep foo.bar
$ # returns nothing
The above curl command results in an HTTP requests that looks like:
GET http://foo.bar/ HTTP/1.1
Host: can-be-anything
User-Agent: curl/7.67.0
Accept: */*
Proxy-Connection: Keep-Alive
ingress-nginx seems to match the request to an ingress based on the hostname of the request-line URI, but doesn't forward this hostname to the backing service.
What you expected to happen:
I expected the Host and X-Forwarded-Host header in the proxied request to always equal the host field of the ingressRule of the ingress resource. I was expecting ingress-nginx to either:
- match the request on the request-line URI with the ingress resource, proxy the request and forward the hostname it used to match the request in one of the headers (X-Forwarded-Host, Host or both)
- match the request on the Host header and return a 404 status code since no such ingress exists
While a request with a full URI in the request-line is not common and not something any webbrowser or HTTP client would normally send, it is part of the HTTP RFC (see section 5.3.2 of RFC7230) and something NGINX supports. I can imagine that this behavior can lead to Host header injections and bad things for applications which rely on and trust the Host and X-Forwarded-Host headers coming from ingress-nginx.
How to reproduce it:
- Install kind
- Install the ingress controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml
- Install echoserver as a backend
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/http-svc.yaml
- Create an ingress
-
echo " apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: foo-bar spec: rules: - host: foo.bar http: paths: - backend: serviceName: http-svc servicePort: 80 path: / " | kubectl apply -f -
-
- Make a request with a URI in the request-line:
-
POD_NAME=$(kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/component=controller -o NAME) kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H "Host: can-be-anything" -x http://127.0.0.1:80/ http://foo.bar/
-
Anything else we need to know:
I'm submitting this as a bug because I expect the Host and X-Forwarded-Host headers to actually tell the backing service something about how the original request was matched to an ingress resource. But, I could see this issue as a feature request if the X-Forwarded-Host header is defined as the header containing the value of the original Host header. In that case I'd like to request a feature to add a header to ingress-nginx which does always contain the host of the ingress which matched the request. And perhaps additionally that the current behavior is clearly documented so that users know not to trust the Host and X-Forwarded-Host header values.
I found that adding the following annotation to the ingress resource causes the Host header in proxied requests to be as I expect it:
nginx.ingress.kubernetes.io/upstream-vhost: $host
I couldn't find a way to alter the X-Forwarded-Host header value.
/kind bug
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
I can still reproduce this issue with Ingress Nginx 0.34.1.
/remove-lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
I can still reproduce this issue with Ingress Nginx 0.41.0.
As for the steps on how to reproduce. Here is a modified ingress YAML for the new networking.k8s.io/v1 API:
echo "
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo-bar
spec:
rules:
- host: foo.bar
http:
paths:
- backend:
service:
name: http-svc
port:
number: 80
path: /
" | kubectl apply -f -
/remove-lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
I can still reproduce this issue with Ingress Nginx 0.44.0.
Turns out the Ingress YAML from my previous comment didn't work as is. I had to add an explicit pathType:
echo "
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo-bar
spec:
rules:
- host: foo.bar
http:
paths:
- backend:
service:
name: http-svc
port:
number: 80
path: /
pathType: Prefix
" | kubectl apply -f -
/remove-lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Reopen this issue or PR with
/reopen - Mark this issue or PR as fresh with
/remove-lifecycle rotten - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/close
@k8s-triage-robot: Closing this issue.
In response to this:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied- After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied- After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closedYou can:
- Reopen this issue or PR with
/reopen- Mark this issue or PR as fresh with
/remove-lifecycle rotten- Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
I'm reopening this issue because it's still bugging me and I think I have an idea of how to fix it.
I believe that this bug could be related to the difference between the nginx variables $host and $http_host. There's a StackOverflow question about it. To summarize, the $http_host variable always contains the Host header of the HTTP request, the $host always contains the hostname as nginx used it to process the request.
If an HTTP request contains a full URL in the request-line then nginx uses the hostname from that URL as a value for $host and uses it to find the matching virtual host. The $http_host variable then simply contains the value of the Host header, which isn't used anywhere else to process the request.
From what I can tell, Ingress-nginx always uses to $http_host variable as a value for the outgoing X-Forwarded-Host header. If I look at the nginx.tmpl file, I see that the $best_http_host variable is set to $http_host and that this variable is used to set the proxied X-Forwarded-Host header. I haven't tested it, but I would guess that changing the set $best_http_host $http_host; line to set $best_http_host $host; would fix this issue. But I can't oversee any potential breakage from that change.
/reopen
@nielsslot: Reopened this issue.
In response to this:
I'm reopening this issue because it's still bugging me and I think I have an idea of how to fix it.
I believe that this bug could be related to the difference between the nginx variables $host and $http_host. There's a StackOverflow question about it. To summarize, the $http_host variable always contains the Host header of the HTTP request, the $host always contains the hostname as nginx used it to process the request.
If an HTTP request contains a full URL in the request-line then nginx uses the hostname from that URL as a value for $host and uses it to find the matching virtual host. The $http_host variable then simply contains the value of the Host header, which isn't used anywhere else to process the request.
From what I can tell, Ingress-nginx always uses to $http_host variable as a value for the outgoing X-Forwarded-Host header. If I look at the nginx.tmpl file, I see that the $best_http_host variable is set to $http_host and that this variable is used to set the proxied X-Forwarded-Host header. I haven't tested it, but I would guess that changing the
set $best_http_host $http_host;line toset $best_http_host $host;would fix this issue. But I can't oversee any potential breakage from that change./reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
@iamNoah1: This request has been marked as needing help from a contributor.
Guidelines
Please ensure that the issue body includes answers to the following questions:
- Why are we solving this issue?
- To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
- Does this issue have zero to low barrier of entry?
- How can the assignee reach out to you for help?
For more details on the requirements of such an issue, please see here and ensure that they are met.
If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.
In response to this:
/triage accepted /help
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The issue has been marked as an important bug and triaged. Such issues are automatically marked as frozen when hitting the rotten state to avoid missing important bugs.
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle frozen
This issue has not been updated in over 1 year, and should be re-triaged.
You can:
- Confirm that this issue is still relevant with
/triage accepted(org members only) - Close this issue with
/close
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted