Support Sync with Kubernetes Secrets [optional secrets-store-csi-driver feature]

[kmadel]

TL;DR

Add support for syncing to Kubernetes Secrets via secretObjects. See https://github.com/kubernetes-sigs/secrets-store-csi-driver#optional-sync-with-kubernetes-secrets

This would allow easier migration of existing pod manifests.

Design

Proposal Azure example: https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/master/test/bats/tests/azure/azure_v1alpha1_secretproviderclass_ns.yaml Vault example: https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/master/test/bats/tests/vault/vault_synck8s_v1alpha1_secretproviderclass.yaml

Alternatives considered No alternative.

Resources Examples:

• Azure example: https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/master/test/bats/tests/azure/azure_v1alpha1_secretproviderclass_ns.yaml

• Vault example: https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/master/test/bats/tests/vault/vault_synck8s_v1alpha1_secretproviderclass.yaml

• Link to documentation

[tam7t]

This should be possible currently, although we don't have an example of it yet:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: app-secrets
spec:
  provider: gcp
  secretObjects:
  - secretName: foosecret
    type: Opaque
    data: 
    - objectName: good1.txt
      key: pwd
  parameters:
    secrets: |
      - resourceName: "projects/$PROJECT_ID/secrets/testsecret/versions/latest"
        fileName: "good1.txt"

This will mount the testsecret to file good1.txt and sync it to a K8S secret called foosecret.

[kmadel]

The missing piece for me was "fileName" getting mapped to "objectName" but I guess that happens here. Tested with the following and it worked when a pod was created using the secretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: cbci-mc-secret-provider
  namespace: cloudbees-core
spec:
  provider: gcp
  secretObjects:
  - secretName: cbci-mc-secret
    type: Opaque
    data: 
    - objectName: token
      key: token
  parameters:
    secrets: |
      - resourceName: "projects/core-workshop/secrets/cbci-workshop-token/versions/latest"
        fileName: "token"

[tam7t]

Thank you for verifying. At a minimum we need to add an integration test + documentation for this.

We could also rename fileName within the parameters to match objectName, but I do feel like fileName is more descriptive.

[wilhelmi]

Do you have an example of your deployment where this worked? I can mount the secret in a volume, but I can't get it to sync to a secret.

[tam7t]

hey @wilhelmi the SecretProviderClass above should work.

If it does not then I suggest taking a look at the logs of the csi-secrets-store daemonset, If you get a message like:

"failed to create Kubernetes secret" err="secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot create resource \"secrets\" in API group \"\" in the namespace \"default\"" spc="default/app-secrets" pod="default/mypod" secret="default/foosecret" spcps="default/mypod-default-app-secrets"

Then you may have missed the install step from https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation.html#alternatively-deployment-using-yamls that allows the csi driver to create secrets:

# If using the driver to sync secrets-store content as Kubernetes Secrets, deploy the additional RBAC permissions
# required to enable this feature
kubectl apply -f deploy/rbac-secretprovidersyncing.yaml

[wilhelmi]

@tam7t That was the ticket! In the case where we just use the secret for an env variable, looks like we need to at least mount the inline-store into a deployment for the sync to take place?

[tam7t]

@wilhelmi Yes, a pod/deployment/etc needs to reference the SecretProviderClass in order to sync the secret using the CSI driver.

[616b2f]

Does this mean when we change the password in e.g. azure keyvault the corresponding k8s Secret object get also updated or is only created once the pod is created and never updated?

[616b2f]

Never mind I found the answer here: https://secrets-store-csi-driver.sigs.k8s.io/topics/secret-auto-rotation.html

TL;DR: you need to aktivate alpha feature via --enable-secret-rotation=true flag.

[joariasl]

If someone is here searching a method to load multiple environment variables from a single Secret in Google Cloud Secret Manager that create a single attribute in the resultant Secret. I've resolved this creating an script to replace the Docker image Entrypoint to load the environments variables from an env file format to the container’s PID 1 scope. https://gist.github.com/joariasl/ed88c2dc556695064dff7d6e89975415 It's possible to test this without set the Docker image using some K8s YAML files like this.

secret-provider-class.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: test-secrets
spec:
  provider: gcp
  # SecretObject defines the desired state of synced K8s secret objects
  secretObjects:
  - secretName: test-secrets
    type: Opaque
    data: 
    - objectName: secrets.env
      key: SECRETS
  parameters:
    secrets: |
      - resourceName: "projects/<your gcp project>/secrets/test-secrets/versions/latest"
        fileName: "secrets.env"

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: envs-from-file
data:
  load-envs-docker-entrypoint.sh: |
    #!/bin/sh

    ENV_PATH_SEPARATOR=${ENV_PATH_SEPARATOR:-':'}

    oldIFS=$IFS
    IFS=$ENV_PATH_SEPARATOR
    for ENV_FILE in $ENV_PATH; do
        if [ -f "$ENV_FILE" ]; then
            export $(grep -v '^#' "$ENV_FILE" | xargs -0)
        else
            echo "$ENV_FILE is not a file" >&2
        fi
    done
    IFS=$oldIFS

    if [ "$IGNORE_EXEC" != "true" ]; then
        # Running params (or CMD) becomes the container’s PID 1
        exec "$@"
    fi

pod.yaml

kind: Pod
apiVersion: v1
metadata:
  name: test-secrets
  labels:
    app.kubernetes.io/name: test-secrets
spec:
  serviceAccountName: test-sa
  containers:
  - name: busybox
    image: k8s.gcr.io/e2e-test-images/busybox:1.29
    command:
    - "/sbin/load-envs-docker-entrypoint.sh"
    - "/bin/sleep"
    - "10000"
    volumeMounts:
    - name: test-secrets
      mountPath: "/mnt/secrets-store"
      readOnly: true
    - name: envs-from-file
      mountPath: /sbin
    env:  
    - name: ENV_PATH
      value: "/mnt/secrets-store/secrets.env"
  volumes:
  - name: test-secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "test-secrets"
  - name: envs-from-file
    configMap:
      name: envs-from-file
      defaultMode: 0555
      items:
      - key: load-envs-docker-entrypoint.sh
        path: load-envs-docker-entrypoint.sh

[aaaaahaaaaa]

Personally I find very cumbersome (and counter-intuitive) the necessity to start a pod in order a the secret to be created. I was hoping for a solution that easily bridges GCP Secret Manager and Kubernetes secrets, without affecting my existing workloads definitions. But I understand that's a limitation coming from the driver itself, not this provider.