vault
vault copied to clipboard
Vault UI ingress can't be configured when HA is enabled - need better documentation on how to do that
Describe the bug
I know it's written in documentation, that when HA is enabled, an ingress is automatically created pointing to vault-active
service. But, it's not specified how can we use ingress for vault-ui
service. Even if I create a separate ingress object just for vault-ui, still it doesn't work in my case.
Ned a better documentation with example to show how to do that. Because, for consul helm chart, there is no such auto pointing to any service, the user defined ingress for consul UI just works fine by pointing to consul-ui service.
To Reproduce Steps to reproduce the behavior:
- Use following values.yaml and install it with helm command:
server:
ha:
enabled: true
replicas: 3
config: |
ui = true
storage "consul" {
path = "vault"
address = "my-consul.mywebsite.com:8500"
}
ingress:
enabled: true
hosts:
- backend:
port:
name: 80
service:
name: vault-ui
host: my-vault.mywebsite.com
pathType: ImplementationSpecific
paths:
- /
ingressClassName: nginx
ui:
enabled: true
- Run
helm install vault hashicorp/vault --set global.name=vault --create-namespace --namespace vault -f values.yaml
- You will see that even though my ingress setup from my vaules.yaml points to vault-ui, final nginx ingress rule created after helm installation points to
vault-active
service and not tovault-ui
. It's Okay, I get that it's clear from documentation. But, how do I achieve what I want to achieve above? - Now, even if I create following ingress object (post above helm installation), still I get
502: Bad Gateway error
when I visit my-vault.mywebsite.com:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: vault-ingress
namespace: vault
spec:
ingressClassName: nginx
rules:
- host: my-vault.mywebsite.com
http:
paths:
- path: /
backend:
serviceName: vault-ui
servicePort: 80
Expected behavior I should be able to see Vault UI when I visit my-vault.mywebsite.com
Environment:
- Vault chart version: vault-0.21.0
- Vault App version: 1.11.2
- Kubectl version:
- Client: Major:“1”, Minor:“21”, GitVersion:“v1.21.10”
- Server: Major:“1”, Minor:“21”, GitVersion:“v1.21.10”
Can someone please help with this? I am sure many other people also might be having the same issue.
This was also asked over at https://discuss.hashicorp.com/t/vault-ingress-how-to-point-to-vault-ui-service-when-ha-is-enabled/43908 where I responded.
I've been a bit confused why the separate ui
section even exists in the Vault helm chart, considering the UI in Vault is served by the main ingress anyway.
After digging in to the Git history, I found an answer - the Vault helm chart was copied from the Consul helm chart originally - and while having a separate UI ingress from Consul makes sense, as you might be routing the Consul API to a client agent - the same is not true for Vault. To me it looks like a vestigial remnant that should have been removed when forking the chart.
I'd suggest removing it from the Vault helm chart now ... except commits since then seem to suggest people have found other ways to make use of it, though I'm not sure of the details.
In any case, this issue probably belongs more in the https://github.com/hashicorp/vault-helm repo than here?
Thanks @maxb I posted the same issue here https://discuss.hashicorp.com/t/vault-ingress-how-to-point-to-vault-ui-service-when-ha-is-enabled/43908 as I was unsure if it is a git issue or not.
I understand from your comment that vault-ui
is a vestigial remnant from helm chart. If that's the case, then what's the correct service we should point our ingress to access vault UI?
What can be next step? How can we use vault UI via ingress when ha is enabled?
Any idea/thoughts? Can someone quickly test this scenario on any k8s setup?
The Vault Helm chart already sets up an ingress. And it works for accessing the UI as well as the API.
The Vault Helm chart already sets up an ingress. And it works for accessing the UI as well as the API.
@maxb That's what doesn't work for me in first place. (If you have any working example, I would be happy to try it out).
The reason being, no matter what you write under ingress section in your values.yaml during setup, as long as you have HA and ingress enabled in values.yaml, it will automatically create an ingress object which points to vault-active
service with port 8200 by completely discarding your user-defined ingress changes.
Here's sentence from their documentation: https://www.vaultproject.io/docs/platform/k8s/helm/configuration#hosts
If ha is enabled the Ingress will point to the active vault server via the active Service.
Also, that doesn't allow you to access vault UI - you get 502: bad gateway
error via nginx ingress controller.
That's the thing that prompt me to come here and open an issue where I created a new separate ingress object but still it doesn't work.
Here's the values.yaml I used for vault setup - the final helm installation throws away my changes and creates an ingress that points to vault-active
as I mentioned earlier - and the same is mentioned in their documentation :
server:
ha:
enabled: true
replicas: 3
config: |
ui = true
storage "consul" {
path = "vault"
address = "my-consul.mywebsite.com:8500"
}
ingress:
enabled: true
hosts:
- backend:
port:
name: 80
service:
name: vault-ui
host: my-vault.mywebsite.com
pathType: ImplementationSpecific
paths:
- /
ingressClassName: nginx
ui:
enabled: true
Hence, I need help with creating
Delete the entire ingress
block and the entire ui
block. That works for me.
Oh, whoops, I'd forgotten that I simply defined a simple ingress outside Helm
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: vault
spec:
rules:
- host: x.y.z
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: vault-active
port:
number: 8200
OK, I deleted my manually-created ingress, and added to the Helm values:
server:
ingress:
enabled: true
hosts:
- host: x.y.z
That results in Helm generating the identical ingress to what I'd previously set up manually
@maxb Sorry for late reply.
Yes, what you are saying that works perfectly fine and I get the UI - I have tested it before too and it works fine.
This is because there is only vault
service created and no vault-active
or no vault-ui
services created with this type of setup.
BUT, I need ingress to work when you enable HA mode
with consul storage. That's what question says in its title.
When we enable HA mode, they automatically create vault-active
and vault-ui
services and no matter what we do, we can't access UI via ingress.
Could you or anyone please test the following setup and check if the UI as accessible with ingress: (Please feel free to modify following vaules.yaml OR create a separate ingress as per your wish BUT keep the HA block as it is in vaules.yaml):
server:
ha:
enabled: true
replicas: 3
config: |
ui = true
storage "consul" {
path = "vault"
address = "my-consul.mywebsite.com:8500"
}
ingress:
enabled: true
hosts:
- backend:
port:
name: 80
service:
name: vault-ui
host: my-vault.mywebsite.com
pathType: ImplementationSpecific
paths:
- /
ingressClassName: nginx
ui:
enabled: true
Please try and let me know if there is any way we can access UI via ingress while HA is enabled.
Change your values.yaml to
server:
ha:
enabled: true
replicas: 3
config: |
ui = true
storage "consul" {
path = "vault"
address = "my-consul.mywebsite.com:8500"
}
ingress:
enabled: true
hosts:
- host: my-vault.mywebsite.com
For me, I had a very similar issue.
It came down to the TCP listener was the issue.
I usually use the setting in my VM based standalone Vaults:
tls_min_version = tlsv3
Found this causes issue within an HA setup in K8s. Have not narrowed it down to Vault or the Ingress. So I had to change it to:
tls_min_version = tlsv2
Now my helm deployment works as expected; even with Helm building the Ingress. Just have to add HTTPS annotation
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
In my case, I am not using any TLS (https) setup. Somehow, it's still not working out.
Really need solution from someone who actually have it working with consul as backend and used ingress for vault.
I provided such an example in my previous comment.
@maxb Sorry for late reply again. I tried your earlier example and also played around with it but somehow it's not working. My consul cluster works just fine on k8s cluster. Still don't know how can I make my vault UI work.
NOTE: One more thing I observed this time that, if I try to initialize vault via CLI (by "exec"ing into the vault pods), I get connection errors between vault servers. So, even via CLI too I am not able to configure vault. May be it is connected to UI not working stuff? Any ideas what I might be doing wrong? My vault version and everything remains same as per I mentioned above in my post.
@neo3matrix - I have faced similar issue today.
I noticed that when setting activeService option to false, Ingress resource will now point to "vault" service instead of "vault-active", and then UI access is working.
server:
ingress:
activeService: false
enabled: true
ingressClassName: "ingress-nginx-vault"
hosts:
- my-vault.mywebsite.com
Then, I saw that "vault-active" service has this selector: vault-active: "true"
When I looked on all 3 HA Vault pods, none of them had this label assigned. So it seems something in our HA setup is broken.
Then I went back to the docs and found this: https://developer.hashicorp.com/vault/docs/platform/k8s/helm/examples/ha-with-raft
After following all steps in this doc (I'm using Raft storage), vault-active: "true" and vault-active: "false" are now assigned to the Vault pods respectively.
Also, I can now see that my cluster has a leader and followers:
kubectl -n vault exec -ti vault-0 -- vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
f46137cb-a31e-004a-c680-dac811dc4af6 vault-0.vault-internal:8201 leader true
206d583a-d7a7-c613-9db1-11c41d134351 vault-1.vault-internal:8201 follower true
63d1d900-a2c3-99d0-c211-1667597ce5a2 vault-2.vault-internal:8201 follower true
Hope this helps.
@tlb1galaxy mentioned to add "HTTPS" work fine for me!
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
Here is my setup
---
# Source: vault-ui/templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: vault-ui
labels:
app.kubernetes.io/name: vault-ui
app.kubernetes.io/instance: vault
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
tls:
- hosts:
- "vault.example.com"
secretName: vault-ui
rules:
- host: "vault.example.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: vault-ui
port:
name: https
---
# Source: vault-ui/templates/certificate.yml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: vault-ui
spec:
dnsNames:
- vault.example.com
secretName: vault-ui
issuerRef:
name: vault-ui
kind: ClusterIssuer
---
# Source: vault-ui/templates/issuer.yml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: vault-ui
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: "[email protected]"
privateKeySecretRef:
name: vault-ui
solvers:
- http01:
ingress:
class: nginx
Addding nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
to ingress annotations helped us get the UI enabled in HA mode. We removed the UI config block and it still worked fine.