Nginx ingress controller overrides x-forwarded-proto even I have used appropriate annotations

[siavashsoleymani]

Describe the bug Nginx ingress controller overrides x-forwarded-proto even when I have used appropriate annotations

To Reproduce This is an overview of what happens in my k8s cluster: User request --> HAproxy (with SSL termination) --> one of the worker nodes which have Nginx ingress controller daemonset --> ingress --> service --> pod I have HAproxy in the edge of my cluster which is responsible for delivering requests to the inside of my cluster and also SSL termination and this is the configuration that I'm using for it (for the sake of shortness I will just put the important part of it):

frontend http bind *:80 bind *:443 ssl crt... mode http redirect scheme https if !{ ssl_fc } default_backend web_servers backend web_servers balance roundrobin default-server check maxconn 20 option forwardfor http-request add-header X-Forwarded-Proto https if { ssl_fc } http-request add-header http_x_forwarded_proto https if { ssl_fc } server server1 192.168.201.197:80 server server2 192.168.201.198:80 server server3 192.168.201.199:80

and I have installed the Nginx Ingress controller in daemonset mode. I have a ingress for my application with this configuration:

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: keycloak-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / nginx.org/hsts-behind-proxy: "True" nginx.org/hsts: "True" nginx.org/location-snippets: | add_header X-Forwarded-Proto https; kubernetes.io/ingress.class: nginx spec: rules: - host: xxxx.xxxx.xxxx.xxx http: paths: - path: / backend: serviceName: keycloak servicePort: 8080

Expected behavior everything works fine and related services and pods are discovered correctly and also when I make a request, the request fulfill correctly by the Nginx ingress controller and delivers to the related pod. The problem is the Keycloak application should know that it is behind a proxy and SSL termination is on the way to serve its web page contents correctly (when it doesn't know the mixed content will render to the browser and browser block the web page ) for this it uses x-forwarded-proto header to determine this, But the Nginx ingress controller overrides this header when it forwards the request to the service. To make sure that Nginx overrides it I have deployed an echo-header application and saw that the Nginx ingress controller overrides the header but when I made the service as NodePort type and directly call it from the HAproxy, the header is there correctly. and as you can see I have used every related annotation to this but not of them worked and I think it is a bug.

Your environment * Version of the Ingress Controller - release version or a specific commit: v1.9.1 * Version of Kubernetes: v1.19.3 * Kubernetes platform (e.g. Mini-kube or GCP): kube-spray * Using NGINX or NGINX Plus: Nginx

Additional context Add any other context about the problem here. Any log files you want to share.

Aha! Link: https://nginx.aha.io/features/IC-102

[pleshakov]

Hi @siavashsoleymani

Could you try to use nginx.org/redirect-to-https annotation? https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#auth-and-ssl-tls

If that annotation is used, NGINX will have a TLS redirect based on X-Forwarded-Proto header (it must be https for no redirect). If there is no redirect, the request will be proxied and the X-Forwarded-Proto to the backend will be set to https.

Yep, there is a limitation that if the TLS redirect is not enabled, then X-Forwarded-Proto will be set to http.

[siavashsoleymani]

Hi @pleshakov I have tried it a few seconds ago and as I guessed didn't work. because the backend application serves on HTTP and when we use the mentioned annotation it expects that application provides HTTPS and it causes another problem in application redirects. Is there any other way to prevent this override? or forcibly add the property to the headers?

[pleshakov]

@siavashsoleymani it is possible to change the template here -- https://github.com/nginxinc/kubernetes-ingress/blob/v1.9.1/internal/configs/version1/nginx.ingress.tmpl#L154 to

proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;

So that in all generated config, the X-Forwarded-Proto is always forwarded to backends

However, that would require using the custom template -- https://github.com/nginxinc/kubernetes-ingress/tree/v1.9.1/examples/custom-templates - you will need to place the whole modified template in the config map

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[Rishats]

Same problem(

[helli0n]

I have the same issues. I have no ability to change the default value of the header. Did somebody find the way how to fix it? Custom template doesn't work as well.

[pleshakov]

@helli0n the only method for now to change that value is to customize the template. To do that customization, it is necessary to put the whole template in the ConfigMap:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
  namespace: nginx-ingress
data:
  ingress-template: |
    # the whole template is inserted here and proxy_set_header X-Forwarded-Proto is updated
        . . .
        proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
        . . .

https://github.com/nginxinc/kubernetes-ingress/tree/v1.11.1/examples/custom-templates

[helli0n]

I tried to create it but can't understand about name of the ConfigMap: name: nginx-config Should I use that name? Can't find the proper configuration file. Read logs type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller

Tried to add ingress-template to the ingress-nginx/ingress-nginx-controller, but didn't help.

When I added

data:
  proxy-connect-timeout: "999"

It was applied to the nginx.conf

ingress-template - doesn't work

[pleshakov]

@helli0n it appears you're using this project https://github.com/kubernetes/ingress-nginx , which is a different Ingress Controller. I suggest referring to its documentation.

[helli0n]

Thank you! I will take a look.

[Entrio]

I am having exactly the same issue as the OP and my setup is the same.

[udayangak]

I am also having the same issue

[tomasfreund]

Would using this config resolve the issue? https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-forwarded-headers It seems that your edge reverse proxy should be adding the X-Forwarded-* headers and the ingress controller is discarding them as it should (for security reasons). Adding the config I mentioned will tell the ingress controller that it is behind a trusted reverse proxy and that it should keep the headers.

[steschuser]

use-forwarded-headers should work, but still failed me

I ended up using proxy-set-headers, pointing to a different config map proxy-set-headers: ingress/custom-headers

and added the header there

apiVersion: v1
data:
  X-Forwarded-Proto: "https"
kind: ConfigMap
metadata:
  name: custom-headers
  namespace: ingress

https://kubernetes.github.io/ingress-nginx/examples/customization/custom-headers/ https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-set-headers

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[belghithahmed]

Describe the bug Nginx ingress controller overrides x-forwarded-proto even when I have used appropriate annotations

To Reproduce This is an overview of what happens in my k8s cluster: User request --> HAproxy (with SSL termination) --> one of the worker nodes which have Nginx ingress controller daemonset --> ingress --> service --> pod I have HAproxy in the edge of my cluster which is responsible for delivering requests to the inside of my cluster and also SSL termination and this is the configuration that I'm using for it (for the sake of shortness I will just put the important part of it):

frontend http
    bind *:80
    bind *:443 ssl crt...
    mode http
    redirect scheme https if !{ ssl_fc }
    default_backend web_servers
backend web_servers
    balance roundrobin
    default-server check maxconn 20
    option forwardfor
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    http-request add-header http_x_forwarded_proto https if { ssl_fc }
    server server1 192.168.201.197:80
    server server2 192.168.201.198:80
    server server3 192.168.201.199:80

and I have installed the Nginx Ingress controller in daemonset mode. I have a ingress for my application with this configuration:

apiVersion: extensions/v1beta1
kind: Ingress
metadata: 
  name: keycloak-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.org/hsts-behind-proxy: "True"
    nginx.org/hsts: "True"
    nginx.org/location-snippets: |
      add_header X-Forwarded-Proto https;
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: xxxx.xxxx.xxxx.xxx
    http: 
     paths:
     - path: /
       backend:
        serviceName: keycloak
        servicePort: 8080

Expected behavior everything works fine and related services and pods are discovered correctly and also when I make a request, the request fulfill correctly by the Nginx ingress controller and delivers to the related pod. The problem is the Keycloak application should know that it is behind a proxy and SSL termination is on the way to serve its web page contents correctly (when it doesn't know the mixed content will render to the browser and browser block the web page ) for this it uses x-forwarded-proto header to determine this, But the Nginx ingress controller overrides this header when it forwards the request to the service. To make sure that Nginx overrides it I have deployed an echo-header application and saw that the Nginx ingress controller overrides the header but when I made the service as NodePort type and directly call it from the HAproxy, the header is there correctly. and as you can see I have used every related annotation to this but not of them worked and I think it is a bug.

Your environment

• Version of the Ingress Controller - release version or a specific commit: v1.9.1
• Version of Kubernetes: v1.19.3
• Kubernetes platform (e.g. Mini-kube or GCP): kube-spray
• Using NGINX or NGINX Plus: Nginx

Additional context Add any other context about the problem here. Any log files you want to share.

Aha! Link: https://nginx.aha.io/features/IC-102

Hi @siavashsoleymani i am facing the same issue Practically i ve the same environment as yours. And i ve also enabled the env proxy_address_forwarding to true on keycloak configuration. And my edge haproxy forward x-forwarded-proto to nginx controller, but i get always mixed co tent issues Could you please tell us how ve you resolved this issue.

[siavashsoleymani]

Hey @belghithahmed, for keycloak we don't use ingress controller we have set node port on keycloak with a fixed port and route request from HAproxy directly to the Keycloak k8s service

[belghithahmed]

Hey @belghithahmed, for keycloak we don't use ingress controller we have set node port on keycloak with a fixed port and route request from HAproxy directly to the Keycloak k8s service

many thanks for your help it works now via nodeport service

[cintoSunny]

Hi everyone, How did you solve this issue? I am doing something similar where I am trying to terminate TLS at the NLB layer. I have an Nginx ingress controller. When I curl to https:, it routes to http://.

curl -ivk https://xx-ssl.g.yy.com/ 
...
< Location: http://xx-ssl.g.yy.com/superset/welcome/
Location: http://xx-ssl.g.yy.com/superset/welcome/
< X-Forwarded-Proto: https
X-Forwarded-Proto: https

If I manually provide the curl -ivk -H "X-Forwarded-Proto: https", it returns https

I have tried all the properties, but still not working. This is what I have in the ingress yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: superseting
  namespace: supersetssltest
  labels:
    app: superset
    chart: superset-0.1.3
    release: superset
    heritage: Helm
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "X-Forwarded-Proto: https";
spec:
  rules:
    - host: xx-ssl.g.yy.com
      http:
        paths:
          - path: /
            backend:
              serviceName: superset
              servicePort: 8080

[MeNsaaH]

@cintoSunny Did you get around yours? I'm struggling to set the X-Forwarded-Proto as well.

[BibbyChung]

@cintoSunny Did you get around yours? I'm struggling to set the X-Forwarded-Proto as well.

for me, add X-Forwarded-Proto it doesn't work.

I change the config.yml of registry to solve it.

// https://docs.docker.com/registry/configuration/
// config.yml

    http:
      relativeurls: true

And somebody said adding environment is working too.

REGISTRY_HTTP_RELATIVEURLS=true

hope saving your time.

[MeNsaaH]

I was able to get mine working using https://github.com/kubernetes/ingress-nginx/issues/6358#issuecomment-1008902987 to set the Headers

Thank you.

[brianehlert]

I was able to get mine working using kubernetes/ingress-nginx#6358 (comment) to set the Headers

Thank you.

FYI - That Issue solution would not be using this project. It refers to specific lua modules that are not used in this project.

[MeNsaaH]

My bad @brianehlert , I must have mistaken this for that project. Thanks for noting that 🙇🏿

[Conspir3D]

The Solution for me was to use the following ingress:

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: apache-nextcloud
    namespace: apache-nextcloud
    annotations:
      nginx.org/proxy-connect-timeout: 3600s
      nginx.org/proxy-read-timeout: 3600s
      nginx.org/proxy-send-timeout: 3600s
      nginx.ingress.kubernetes.io/proxy-body-size: "8g"
      nginx.ingress.kubernetes.io/configuration-snippet: |
        more_set_headers "Host              $http_host";
        more_set_headers "X-Real-IP         $remote_addr";
        more_set_headers "X-Forwarded-Proto $scheme";
        more_set_headers "X-Forwarded-For   $proxy_add_x_forwarded_for";

  spec:
.
.
.

[brianehlert]

You are mixing annotations between two different NGINX Ingress Controller projects. nginx.org is this project and nginx.ingress.kubernetes.io is the community nginx ingress controller. more_set_headers is actually specific to OpenResty which is also under the hood of the community ingress controller, or could be added to this project (but this project would use a different snippet annotation)