Troubleshoot issue with whitelist in Azure vm

[psapozh]

Hello

We are using ingress controller version 1.6.0 inside Azure AKS. I am trying to add ingress resource that has a certain host, path and also whitelist annotation. I am setting this whitelist value to be the public ip of one of our Azure vms. What happens is from my local I try to initiate this request and I get 403 - this is expected correct behaviour because my local ip is not one of the whitelisted ips. However, then I try to initiate same request from this vm whos public ip is the one I whitelisted and from there I get 403 as well. Not sure what I am doing wrong here. I know having whitelist there does something because if I comment it out then it allows that path from all ips. Also, I dont know how to debug this. Tried looking at logs of the pods nothing obvious there.

[ivanmatmati]

Hi @psapozh , can you post your white list configuration and the resulting haproxy configuration file section with the deny clause.

[psapozh]

ok so I have actually upgraded the installation to the latest version.

But here's the Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: interopswaggerui
  namespace: interop
  annotations:
      meta.helm.sh/release-name: interopapideployment
      meta.helm.sh/release-namespace: interop
      haproxy.org/whitelist: "x.x.x.x"
      ingress.class: "haproxy"
  labels:
    app.kubernetes.io/instance: interopapideployment
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: interopapideployment
    app.kubernetes.io/version: 1.16.0
    helm.sh/chart: interopapideployment-0.1.0
spec:
  rules:
  - host: someHost.com
    http:
      paths:
        - path: /swagger-ui
          backend:
            serviceName: interopapideployment
            servicePort: 8080

and this is a snippet from haproxy.cfg

frontend http
  mode http
  bind 0.0.0.0:80 name v4
  bind :::80 name v6
  http-request set-var(txn.base) base
  http-request set-var(txn.path) path
  http-request set-var(txn.host) req.hdr(Host),field(1,:),lower
  http-request set-var(txn.host_match) var(txn.host),map(/etc/haproxy/maps/host.map)
  http-request set-var(txn.host_match) var(txn.host),regsub(^[^.]*,,),map(/etc/haproxy/maps/host.map,'') if !{ var(txn.host_match) -m found }
  http-request set-var(txn.path_match) var(txn.host_match),concat(,txn.path,),map(/etc/haproxy/maps/path-exact.map)
  http-request set-var(txn.path_match) var(txn.host_match),concat(,txn.path,),map_beg(/etc/haproxy/maps/path-prefix.map) if !{ var(txn.path_match) -m found }
  http-request deny deny_status 403 if { var(txn.path_match) -m dom 47eee10fc68d3a2e771e64481f8b735c } !{ src -f /etc/haproxy/maps/whitelist-f41f9c053c84b81407234bb7a7b30318.map }
  use_backend %[var(txn.path_match),field(1,.)]
  default_backend ingress_haproxytech-kubernetes-ingress-default-backend_http

[ivanmatmati]

Sounds good. Can you check that you've got your IP in /etc/haproxy/maps/whitelist-f41f9c053c84b81407234bb7a7b30318.map ? You can also check that 47eee10fc68d3a2e771e64481f8b735c is indeed in /etc/haproxy/maps/path-exact.map or /etc/haproxy/maps/path-prefix.map. I suppose the correct Host header was provided in your previous attempt.

[psapozh]

Confirmed the contents of whitelist is the correct ip. The path-exact contains the correct host and also this:

interop_interopapideployment_http.47eee10fc68d3a2e771e64481f8b735c.47eee10fc68d3a2e771e64481f8b735c

path-prefix has exactly same contents as path-exact.

[psapozh]

Also entering blacklist with my ip has no effect. I am able to reach the page from my ip. Entering blacklist and whitelist at the same time, seems that whitelist is taking more priority over blacklist and blocking from everywhere regardless of the ips entered.

[psapozh]

Hello

May I get any update on this please?

[ivanmatmati]

Hi @psapozh , can you check you get something like this in your frontends configuration:

http-request deny deny_status 403 if { src -f /xxx/maps/blacklist-xxxx.map }
http-request deny deny_status 403 if !{ src -f /xxx/maps/whitelist-xxxx.map }

If so check the contents in these files. As you see the behavior is to deny first by blacklist and then check whitelist.

[psapozh]

Hi @ivanmatmati

Yes this exists in haproxy.cfg file under both frontend http and https.

This is the exact line:

http-request deny deny_status 403 if { var(txn.path_match) -m dom 586b723fa6ddfa0bcaa67f6e549e7361 } !{ src -f /etc/haproxy/maps/whitelist-cd88bc74775290333e2295dfaf712158.map }

However, I think the root of the problem is the fact that haproxy pod sees the request coming in as ip of the node on which the pod runs not the actual client ip. I have tried to solve this by setting externalTrafficPolicy=Local on haproxy LoadBalancer service and this didn't help. Also, I tried setting forwarded-for: "true" but that also doesn't seem to do anything.

[ivanmatmati]

forwarded-for should do the job. Can you check you get option forwardfor in your corresponding backend section ?

[psapozh]

Hi @ivanmatmati

Yes, I have multiple backend sections one for haproxy ingress itself and one for my app ingress

In both sections I have option forwardfor. The other thing is right now I have both enabled this and externalTrafficPolicy

In my yaml I have:

request-capture: |
      hdr(X-Forwarded-Proto)
      hdr(X-Forwarded-For)
      hdr(X-Forward-For)

this should print out that header value as part of log but it's coming empty unless I set that header myself in postman and also only if the whitelisting rule is not there. Otherwise even if I set and whitelisting is there its coming empty.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.