Provider Failing to Bypass WAF Rules Using HTTP Header

[raynopus]

I have deployed ArgoCD to an Amazon EKS cluster and used ALB Controller and External DNS to bring traffic into the cluster to access the ArgoCD web UI. The ALB is linked to a web ACL that blocks requests unless it is either from a whitelisted IP address or the request contains an X-WAF-Bypass HTTP header with a secret value.

provider "argocd" {
  server_addr = "argocd.REDACTED"
  username    = "admin"
  password    = data.kubernetes_secret.argocd_admin.data.password
  headers     = ["X-WAF-Bypass: REDACTED"]
}

resource "argocd_cluster" "REDACTED" {
  server = data.aws_eks_cluster.REDACTED.endpoint
  name   = "REDACTED"

  config {
    aws_auth_config {
      cluster_name = "REDACTED"
      role_arn     = data.aws_iam_role.REDACTED.arn
    }
    tls_client_config {
      ca_data = base64decode(data.aws_eks_cluster.REDACTED.certificate_authority[0].data)
    }
  }
}

My workstation's IP address is in the whitelist so everything works fine there. However, from GitHub Actions, the ArgoCD Terraform provider encounters the following error. Presumably because it is not making it past the web ACL.

│ Error: failed to create new session
│ 
│   with argocd_cluster.REDACTED,
│   on REDACTED.tf line [55](https://github.com/REDACTED), in resource "argocd_cluster" "REDACTED":
│   55: resource "argocd_cluster" "REDACTED" {
│ 
│ rpc error: code = Unknown desc = unexpected HTTP status code received from
│ server: 464 (); malformed header: missing HTTP content-type

This is odd because none of my web ACL rules respond with 464. I removed my workstation IP address from the whitelist so I now see the same behavior there. From my workstation, curl https://argocd.REDACTED gets a 403 response as defined in my web ACL rules, and curl -H "X-WAF-Bypass: REDACTED" https://argocd.REDACTED gets 200. So the WAF is working correctly. There seems to be something off with how the ArgoCD Terraform provider is handling the custom HTTP headers that is causing the WAF to misbehave. If I remove the web ACL entirely then everything works fine.

Terraform Version, ArgoCD Provider Version and ArgoCD Version

Terraform version: 1.11.2
ArgoCD provider version: 7.5.2
ArgoCD version: v2.14.9+38985bd

Expected Behavior

The ArgoCD Terraform provider should be setting the X-WAF-Bypass HTTP header to allow requests past my web ACL rules.

Actual Behavior

The ArgoCD Terraform provider is receiving response code 464 even though that response code is not from my web ACL rules. It seems that Amazon WAF decides it doesn't like the headers before it reaches my rules.

Important Factoids

References

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[raynopus]

I would love to get a response ...

[the-technat]

Hi @raynopus, thanks for pinging us again. As far as I understand your case you suspect the Terraform provider doesn't set the HTTP headers correctly that you passed in and thus it's blocked by your WAF?

If so, could you maybe check if a Terraform run using TF_LOG=trace shows the HTTP headers that the provider sends out? So that we might be able narrow down the bug?

I'm also trying to reproduce the same on my side as well in the meantime.

[the-technat]

@raynopus I tried spinning up an environment to check whether the provider sets headers correctly. I used mitmproxy for this in reserve-proxy mode and I suspect your WAF isn't GRPC aware.

The provider by default communicates over GRPC with the Argo CD API, omitting any standard HTTP headers:

If you enable grpc_web = true in the provider configuration, GRPC is tunneled over HTTP somehow and then HTTP headers are set as expected:

Could you try using grpc_web = true and report if that solves your issue?

[VirtualEvan]

I am seeing a similar behavior with the headers configuration.

In my case, I am trying to configure the provider using the IP Address of my ingress and set the Host header separately. It however, returns a 404 as if the header was not set.

provider "argocd" {
  server_addr = "192.168.1.250:443"
  username    = "admin"
  password    = data.kubernetes_secret.argocd_admin.data.password
  insecure    = true
  grpc_web    = true
  headers     = [
    "Host: argocd.REDACTED"
  ]
}

Using curl, I am able to get a response. curl https://192.168.1.250:443/session.SessionService/Create -H "Host: argocd.REDACTED" --insecure

I tried using mitmproxy as @the-technat did, but here it worked fine. The proxy recognized the header and forwarded the request, so terraform succeeded in contacting argocd. I don't know what conclusions to draw from that.

[the-technat]

@VirtualEvan I have similar results when trying to connect to Argo CD over ingress-nginx and setting the Host header. In which mode did you run mitmproxy to analyze? Maybe I'm stupid but somehow I can't get it to work with regular HTTP proxy env variables, it seems like our provider doesn't pick them up (despite using regular go http libraries that should support it). And running it in reverse-proxy mode seems to make things even more complex...

[VirtualEvan]

@the-technat Reverse proxy. Yesterday was my first time using this tool, so I went with mitmweb for simplicity. However, I just tested it with the cli app and got the same results. I didn't try anything that included env variables 😕

mitmproxy

mitmproxy --mode reverse:https://argocd.REDACTED@8443 --ssl-insecure

mitmweb

Capture > Reverse Proxy > Enable Forward HTTPS traffic to "argocd.REDACTED" > ⚙️ > Listen Host * (default) | Listen Port 8443 Options > Do not verify server certificates (I am using self-signed certs)

Provider

provider "argocd" {
  server_addr = "127.0.0.1:8443"
  username    = "admin"
  password    = data.kubernetes_secret.argocd_admin.data.password
  insecure    = true
  grpc_web    = true
  headers = [
    "Host: argocd.REDACTED"
  ]
}

resource "argocd_repository" "harbor" {
  name       = "harbor"
  repo       = "harbor.REDACTED"
  type       = "helm"
  insecure   = true
  enable_oci = true
}

I am using Traefik as Ingress Controller

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.