Helm chart: Allow referencing secrets for s3 storage configuration

[tchinmai7]

Is your feature request related to a problem? Please describe. The storage configuration for loki's helm chart currently requires users to pass in the secretKey and accessKey as plain-text in the values. This makes gitops driven installations difficult, as secrets must either be injected by a plugin OR stored in values (bad)

Describe the solution you'd like Allow passing in secretRef's for the accessKey and SecretKey

[amannagpal2]

how you configured accesskey then?

[sebhoss]

We use kustomize exec functions to modify the configmap and inject secrets like this:

1. Create kustomization.yaml like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ...
transformers:
  - transform-configmap-loki.yaml # the important part here

2. Create the transformer like this:

apiVersion: example.com/v1
kind: Transformer
metadata:
  name: loki
  namespace: loki
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: kustomize-exec-functions/configmap-loki.sh
spec:
  secret-path: cluster.example/minio/tenants/loki # path in our secret store that contains credentials

3. Write the exec function:

#!/usr/bin/env sh

resourceList=$(cat) # read the `kind: ResourceList` from stdin

kind='ConfigMap'
name=$(echo "${resourceList}" | yq '.functionConfig.metadata.name' -)
namespace=$(echo "${resourceList}" | yq '.functionConfig.metadata.namespace' -)
secret_path=$(echo "${resourceList}" | yq '.functionConfig.spec.secret-path' -)

access_key=$(read-secret-from-store "${secret_path}" access-key)
secret_key=$(read-secret-from-store "${secret_path}" secret-key)

echo "${resourceList}" |
  yq "
   ( .items[]
    | select(.kind == \"${kind}\" and .metadata.name == \"${name}\" and .metadata.namespace == \"${namespace}\")
    | .data[\"config.yaml\"])
    |= (from_yaml
      | .common.storage.s3.access_key_id = \"${access_key}\"
      | .common.storage.s3.secret_access_key = \"${secret_key}\"
      | .ruler.storage.s3.access_key_id = \"${access_key}\"
      | .ruler.storage.s3.secret_access_key = \"${secret_key}\"
    | to_yaml)"

Kustomize will send all existing resources as a ResourceList into this function and we modify the ConfigMap named 'loki' on the fly and output the result.

In the future, kustomize might get the ability to modify those nested structures values directly once https://github.com/kubernetes-sigs/kustomize/issues/4517 lands.

[mrclrchtr]

In my opinion, this feture is a must-have for implementing git ops in a meaningful way. I would be very happy if it were implemented.

Nevertheless, thanks for the workaround, I will try it out.

[evilhamsterman]

Not just for GitOps, it should be default option that secrets should be stored as Kubernetes secrets. This may require more than just changes to the Helm chart, but the application itself to load secrets from the pod environment.

I'm surprised that a company like Grafana wouldn't consider this

[evilhamsterman]

So it's not documented unless you look at the values file. But you can use the environment variable substitution feature https://grafana.com/docs/loki/latest/configure/#use-environment-variables-in-the-configuration

You have to set

global:
  extraArgs:
    - -config.expand-env=true
   extraEnvFrom:
    - secretRef:
        name: <secretname>
loki:
  storage:
    s3:
      secretAccessKey: "${SECRET_ENV_VAR}"

Then the values in the ConfigMap will be the ${VARIABLE} format, Kubernetes will inject the secrets as Environment Variables and the app will expand them into the config before load

[evilhamsterman]

Turns out the global settings are not picked up by several components currently so you have to specify the envfrom on each component