Cannot source secrets injected from Vault

[DSamuylov]

I have the latest Grafana on my Kubernetes cluster deployed using kube-prometheus-stack (version 36.2.0), and I would like to inject secrets, e.g. SMTP username and password, directly from Vault by HashiCorp following this example.

My values.yaml includes the following specification:

  serviceAccount:
    create: true
    name: grafana-app
    autoMount: true
  podAnnotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "grafana-app"
    vault.hashicorp.com/agent-pre-populate-only: "true"
    vault.hashicorp.com/agent-inject-secret-config-grafana-smtp.env: "monitoring/data/grafana/smtp"
    vault.hashicorp.com/agent-inject-template-config-grafana-smtp.env: |
      {{ with secret "monitoring/data/grafana/smtp" }}
        export GF_SMTP_USER_TEST="{{ .Data.data.username }}"
        export GF_SMTP_PASSWORD_TEST="{{ .Data.data.password }}"
      {{ end }}
  command:
    - /bin/sh
    - -c
    - source /vault/secrets/config-grafana-smtp.env && /run.sh

(NOTE: I add _TEST suffix to avoid the clash with env variables that I inject using envValueFrom from a secreted stored on Kubernetes.)

When I connect to the Grafana container using:

kubectl exec -it POD_NAME -n monitoring -c grafana -- sh

and I print all environment variables using env | grep GF_SMTP, I do not see GF_SMTP_USER_TEST and GF_SMTP_PASSWORD_TEST.

However, I can see that the file injected by Vault exists ls /vault/secrets/config-grafana-smtp.env.

Is it a bug or I do not source the injected file correctly?

[DSamuylov]

I would highly appreciate if somebody could provide any feedback on this issue.

[ajfriesen]

I am looking into this as well, did you find a suitable solution?

I tried to set the env variables for username and password myself:

GF_SECURITY_ADMIN_PASSWORD
GF_SECURITY_ADMIN_USER

with the helm values like so:

podAnnotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/agent-inject-secret-helloworld: "secret/helloworld"
  vault.hashicorp.com/role: "grafana"
  vault.hashicorp.com/agent-inject-template-foo: |
    {{- with secret "secret/helloworld" -}}
      export GF_SECURITY_ADMIN_USER="{{ .Data.helloworld.username }}"
      export GF_SECURITY_ADMIN_PASSWORD="{{ .Data.helloworld.password }}"
    {{- end }}

But when I check my container it seems that grafana is still creating a random password.

Any idea how to remove the automatic password creation? Did not find where this is happening yet.

[DSamuylov]

I found a work around though reading the username and password from secrets:

grafana:
  envValueFrom:
    GF_SMTP_USER:
      secretKeyRef:
        name: smtp-secret
        key: username
    GF_SMTP_PASSWORD:
      secretKeyRef:
        name: smtp-secret
        key: password

However, it sounds like a work around and potentially less secure. If you find a solution, please share!

[ajfriesen]

This is really just a workaround.

I want to set the username and password for the initial admin user from vault values. But somehow grafana always creates the password automatically.

Also, I do not want to use Kubernetes secrets, because vault is already the secret store. The purpose of vault is to not get secrets into kubernetes/etcd.

[DSamuylov]

Completely agree with it! Hopefully we can get an answer from the maintainers of the repository on how to address it.

[DSamuylov]

@ajfriesen did you manage to resolve this issue? I am looking into it again, and I would highly appreciate if you could share your status on it.

[bincyber]

I have not tested this, but try overriding the admin settings in grafana.ini using the Env provider

grafana:
  grafana.ini:
    security:
      admin_user: "$__env{GF_SECURITY_ADMIN_USER}"
      admin_password: "$__env{GF_SECURITY_ADMIN_PASSWORD}"

[Migueljfs]

@bincyber that does not work, I'm on a similar situation and tried that exactly with no success.

Don't think the helm chart supports setting the password through the values directly, only through a kubernetes secret

[ajfriesen]

@DSamuylov

Finally had some time to work on this and I think I have a solution.

Create a secret in Vault called monitoring/grafana with the Key value pair GRAFANA_ADMIN_PASSWORD=yoursuperpassword

Create a policy to allow reading that secrets:

path "monitoring/data/grafana" {
  capabilities = ["read"]
}

Create a role that binds the service account and the namespace in the vault.

And finally the Grafana values.yaml:

This will create a file /vault/secrets/environmental-variables-config.txt in the pod which is templated via vault:

export GF_SECURITY_ADMIN_PASSWORD=yoursuperpassword

You can then overwrite the command and args section to first source the file and the then the default /run.sh to start the docker container.

The GF_SECURITY_ADMIN_PASSWORD is from the docs here: https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#override-configuration-with-environment-variables

serviceAccount:
  create: true
  name: grafana

podAnnotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/agent-inject-status: "update"
  vault.hashicorp.com/agent-inject-secret-environmental-variables-config.txt: "monitoring/data/grafana"
  vault.hashicorp.com/role: "monitoring"
  vault.hashicorp.com/agent-inject-template-environmental-variables-config.txt: |
      {{ with secret "monitoring/data/grafana" -}}
      export GF_SECURITY_ADMIN_PASSWORD={{ .Data.data.GRAFANA_ADMIN_PASSWORD }}
      {{ end }}

command:
- "/bin/bash"
- "-c"

args:
  - "source /vault/secrets/environmental-variables-config.txt && /run.sh"

Only downside is: This will still create the kubernetes secret with the password. However, you could remove this with a kustomize post-renderer but I am okay with that.

[jwhy89]

Running into the same issue. One thing to consider is that the grafana deploment pod is using either the it's own service account or the default one if you disable the service account creation. For the pod to grab the Vault secret, wouldn't we also want to be able to select the serviceAccount to be used with the grafana pod?