Vault UI ingress can't be configured when HA is enabled - need better documentation on how to do that

[neo3matrix]

Describe the bug I know it's written in documentation, that when HA is enabled, an ingress is automatically created pointing to vault-active service. But, it's not specified how can we use ingress for vault-ui service. Even if I create a separate ingress object just for vault-ui, still it doesn't work in my case.

Ned a better documentation with example to show how to do that. Because, for consul helm chart, there is no such auto pointing to any service, the user defined ingress for consul UI just works fine by pointing to consul-ui service.

To Reproduce Steps to reproduce the behavior:

1. Use following values.yaml and install it with helm command:

server:
  ha:
    enabled: true
    replicas: 3
    config: |
      ui = true
      storage "consul" {
        path = "vault"
        address = "my-consul.mywebsite.com:8500"
      }
  ingress:
    enabled: true
    hosts:
    - backend:
        port:
          name: 80
        service:
          name: vault-ui
      host: my-vault.mywebsite.com
      pathType: ImplementationSpecific
      paths:
      - /
    ingressClassName: nginx
ui:
  enabled: true

2. Run helm install vault hashicorp/vault --set global.name=vault --create-namespace --namespace vault -f values.yaml
3. You will see that even though my ingress setup from my vaules.yaml points to vault-ui, final nginx ingress rule created after helm installation points to vault-active service and not to vault-ui. It's Okay, I get that it's clear from documentation. But, how do I achieve what I want to achieve above?
4. Now, even if I create following ingress object (post above helm installation), still I get 502: Bad Gateway error when I visit my-vault.mywebsite.com:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: vault-ingress
  namespace: vault
spec:
  ingressClassName: nginx
  rules:
  - host: my-vault.mywebsite.com
    http:
      paths:
      - path: /
        backend:
          serviceName: vault-ui
          servicePort: 80

Expected behavior I should be able to see Vault UI when I visit my-vault.mywebsite.com

Environment:

• Vault chart version: vault-0.21.0
• Vault App version: 1.11.2
• Kubectl version:

• Client: Major:“1”, Minor:“21”, GitVersion:“v1.21.10”
• Server: Major:“1”, Minor:“21”, GitVersion:“v1.21.10”

Can someone please help with this? I am sure many other people also might be having the same issue.

[maxb]

This was also asked over at https://discuss.hashicorp.com/t/vault-ingress-how-to-point-to-vault-ui-service-when-ha-is-enabled/43908 where I responded.

I've been a bit confused why the separate ui section even exists in the Vault helm chart, considering the UI in Vault is served by the main ingress anyway.

After digging in to the Git history, I found an answer - the Vault helm chart was copied from the Consul helm chart originally - and while having a separate UI ingress from Consul makes sense, as you might be routing the Consul API to a client agent - the same is not true for Vault. To me it looks like a vestigial remnant that should have been removed when forking the chart.

I'd suggest removing it from the Vault helm chart now ... except commits since then seem to suggest people have found other ways to make use of it, though I'm not sure of the details.

In any case, this issue probably belongs more in the https://github.com/hashicorp/vault-helm repo than here?

[neo3matrix]

Thanks @maxb I posted the same issue here https://discuss.hashicorp.com/t/vault-ingress-how-to-point-to-vault-ui-service-when-ha-is-enabled/43908 as I was unsure if it is a git issue or not.

I understand from your comment that vault-ui is a vestigial remnant from helm chart. If that's the case, then what's the correct service we should point our ingress to access vault UI? What can be next step? How can we use vault UI via ingress when ha is enabled?

Any idea/thoughts? Can someone quickly test this scenario on any k8s setup?

[maxb]

The Vault Helm chart already sets up an ingress. And it works for accessing the UI as well as the API.

[neo3matrix]

The Vault Helm chart already sets up an ingress. And it works for accessing the UI as well as the API.

@maxb That's what doesn't work for me in first place. (If you have any working example, I would be happy to try it out). The reason being, no matter what you write under ingress section in your values.yaml during setup, as long as you have HA and ingress enabled in values.yaml, it will automatically create an ingress object which points to vault-active service with port 8200 by completely discarding your user-defined ingress changes.
Here's sentence from their documentation: https://www.vaultproject.io/docs/platform/k8s/helm/configuration#hosts

If ha is enabled the Ingress will point to the active vault server via the active Service.

Also, that doesn't allow you to access vault UI - you get 502: bad gateway error via nginx ingress controller.

That's the thing that prompt me to come here and open an issue where I created a new separate ingress object but still it doesn't work.

Here's the values.yaml I used for vault setup - the final helm installation throws away my changes and creates an ingress that points to vault-active as I mentioned earlier - and the same is mentioned in their documentation :

server:
  ha:
    enabled: true
    replicas: 3
    config: |
      ui = true
      storage "consul" {
        path = "vault"
        address = "my-consul.mywebsite.com:8500"
      }
  ingress:
    enabled: true
    hosts:
    - backend:
        port:
          name: 80
        service:
          name: vault-ui
      host: my-vault.mywebsite.com
      pathType: ImplementationSpecific
      paths:
      - /
    ingressClassName: nginx
ui:
  enabled: true

Hence, I need help with creating

[maxb]

Delete the entire ingress block and the entire ui block. That works for me.

[maxb]

Oh, whoops, I'd forgotten that I simply defined a simple ingress outside Helm

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: vault
spec:
  rules:
    - host: x.y.z 
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: vault-active
                port:
                  number: 8200

[maxb]

OK, I deleted my manually-created ingress, and added to the Helm values:

server:
  ingress:
    enabled: true
    hosts:
      - host: x.y.z

That results in Helm generating the identical ingress to what I'd previously set up manually

[neo3matrix]

@maxb Sorry for late reply. Yes, what you are saying that works perfectly fine and I get the UI - I have tested it before too and it works fine. This is because there is only vault service created and no vault-active or no vault-ui services created with this type of setup.

BUT, I need ingress to work when you enable HA mode with consul storage. That's what question says in its title. When we enable HA mode, they automatically create vault-active and vault-ui services and no matter what we do, we can't access UI via ingress.

Could you or anyone please test the following setup and check if the UI as accessible with ingress: (Please feel free to modify following vaules.yaml OR create a separate ingress as per your wish BUT keep the HA block as it is in vaules.yaml):

server:
  ha:
    enabled: true
    replicas: 3
    config: |
      ui = true
      storage "consul" {
        path = "vault"
        address = "my-consul.mywebsite.com:8500"
      }
  ingress:
    enabled: true
    hosts:
    - backend:
        port:
          name: 80
        service:
          name: vault-ui
      host: my-vault.mywebsite.com
      pathType: ImplementationSpecific
      paths:
      - /
    ingressClassName: nginx
ui:
  enabled: true

Please try and let me know if there is any way we can access UI via ingress while HA is enabled.

[maxb]

Change your values.yaml to

server:
  ha:
    enabled: true
    replicas: 3
    config: |
      ui = true
      storage "consul" {
        path = "vault"
        address = "my-consul.mywebsite.com:8500"
      }
  ingress:
    enabled: true
    hosts:
    - host: my-vault.mywebsite.com

[tlb1galaxy]

For me, I had a very similar issue.
It came down to the TCP listener was the issue. I usually use the setting in my VM based standalone Vaults:

tls_min_version = tlsv3

Found this causes issue within an HA setup in K8s. Have not narrowed it down to Vault or the Ingress. So I had to change it to:

tls_min_version = tlsv2

Now my helm deployment works as expected; even with Helm building the Ingress. Just have to add HTTPS annotation

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

[neo3matrix]

In my case, I am not using any TLS (https) setup. Somehow, it's still not working out.

Really need solution from someone who actually have it working with consul as backend and used ingress for vault.

[maxb]

I provided such an example in my previous comment.

[neo3matrix]

@maxb Sorry for late reply again. I tried your earlier example and also played around with it but somehow it's not working. My consul cluster works just fine on k8s cluster. Still don't know how can I make my vault UI work.

NOTE: One more thing I observed this time that, if I try to initialize vault via CLI (by "exec"ing into the vault pods), I get connection errors between vault servers. So, even via CLI too I am not able to configure vault. May be it is connected to UI not working stuff? Any ideas what I might be doing wrong? My vault version and everything remains same as per I mentioned above in my post.

[moshiaiz]

@neo3matrix - I have faced similar issue today.

I noticed that when setting activeService option to false, Ingress resource will now point to "vault" service instead of "vault-active", and then UI access is working.

server:
  ingress:
    activeService: false
    enabled: true
    ingressClassName: "ingress-nginx-vault"
    hosts:
      - my-vault.mywebsite.com

Then, I saw that "vault-active" service has this selector: vault-active: "true"

When I looked on all 3 HA Vault pods, none of them had this label assigned. So it seems something in our HA setup is broken.

Then I went back to the docs and found this: https://developer.hashicorp.com/vault/docs/platform/k8s/helm/examples/ha-with-raft

After following all steps in this doc (I'm using Raft storage), vault-active: "true" and vault-active: "false" are now assigned to the Vault pods respectively.

Also, I can now see that my cluster has a leader and followers:

kubectl -n vault exec -ti vault-0 -- vault operator raft list-peers
Node                                    Address                        State       Voter
----                                    -------                        -----       -----
f46137cb-a31e-004a-c680-dac811dc4af6    vault-0.vault-internal:8201    leader      true
206d583a-d7a7-c613-9db1-11c41d134351    vault-1.vault-internal:8201    follower    true
63d1d900-a2c3-99d0-c211-1667597ce5a2    vault-2.vault-internal:8201    follower    true

Hope this helps.

[tunchamroeun]

@tlb1galaxy mentioned to add "HTTPS" work fine for me!

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

Here is my setup

---
# Source: vault-ui/templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: vault-ui
  labels:
    app.kubernetes.io/name: vault-ui
    app.kubernetes.io/instance: vault
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  tls:
    - hosts:
        - "vault.example.com"
      secretName: vault-ui
  rules:
    - host: "vault.example.com"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: vault-ui
                port:
                  name: https
---
# Source: vault-ui/templates/certificate.yml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: vault-ui
spec:
  dnsNames:
    - vault.example.com
  secretName: vault-ui
  issuerRef:
    name: vault-ui
    kind: ClusterIssuer
---
# Source: vault-ui/templates/issuer.yml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-ui
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: "[email protected]"
    privateKeySecretRef:
      name: vault-ui
    solvers:
      - http01:
          ingress:
            class: nginx

[rajeshvari83]

Addding nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" to ingress annotations helped us get the UI enabled in HA mode. We removed the UI config block and it still worked fine.