Secret creation making helm upgrade failure

[vivuu1989]

We configured csi driver in our cluster for secret management and used below secret providerclass template to automatically assign secrets to the deployments env variable. The above setup is working fine.

But 2 things where i have issue. Whenever new changes done to the secret, say if added a new secret to the yaml and keyvault, the next release will fail with helm upgrade command, stating speified secret is not find.

So inorder to solve this, I have to uninstall all helm release and need to install the helm release again, which meant down time. How I can I achieve this scenario without any down time.

Secondly, is there any recommended way to restart the Pods when the secret template changes:

values.yaml for MyAppA

keyvault:
  name: mykv
  tenantId: ${tenantId}$
  clientid: "#{spid}#"
  clientsecret: "#{spsecret}#"
  secrets:
    - MyAPPA_SECRET1_NAME1
    - MyAPPA_SECRET2_NAME2
    - MyAPPA_SECRET3_NAME3

deployment.yaml, ENV part is as below

    {{- if eq .Values.keyvault.enabled true }}
    {{- range .Values.keyvault.secrets }}{{/* <-- only one range loop */}}
      - name: {{ . }}
        valueFrom:
          secretKeyRef:
            name: {{ $.Release.Name }}-kvsecret
            key: {{ . }}
      {{- end }}
      {{- end }}
      volumeMounts: 
      - name:  {{ $.Release.Name }}-volume
        mountPath: '/mnt/secrets-store'
        readOnly: true
  volumes:
    - name: {{ $.Release.Name }}-volume
      csi:
        driver: 'secrets-store.csi.k8s.io'
        readOnly: true
        volumeAttributes:
          secretProviderClass: {{ $.Release.Name }}-secretproviderclass
        nodePublishSecretRef:
          name: {{ $.Release.Name }}-secrets-store-creds
		  
		  

secretProviderClass yaml file is as below.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: {{ $.Release.Name }}-secretproviderclass
  labels:
    app: {{ $.Release.Name }}
    chart: "{{ $.Release.Name }}-{{ .Chart.Version }}"
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  provider: azure
  secretObjects:
  - data:
      {{- range .Values.keyvault.secrets }}{{/* <-- only one range loop */}}
    - key: {{ . }}
      objectName: {{ $.Release.Name | upper }}-{{ . }} 
      {{- end }}      
    secretName: {{ $.Release.Name }}-kvsecret
    type: opaque
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    userAssignedIdentityID: ""
    keyvaultName: {{ .Values.keyvault.name | default "mydev-kv" }}   
    objects: |
      array:
        {{- range .Values.keyvault.secrets }}{{/* <-- only one range loop */}}        
        - |
          objectName: {{ $.Release.Name | upper }}-{{ . }}
          objectType: secret
        {{- end }}
    tenantId: {{ .Values.keyvault.tenantid }}
{{- end }}
{{- end -}}
{{- define "commonobject.secretproviderclass" -}}
{{- template "commonobject.util.merge" (append . "commonobject.secretproviderclass.tpl") -}}
{{- end -}}

[aramase]

So inorder to solve this, I have to uninstall all helm release and need to install the helm release again, which meant down time. How I can I achieve this scenario without any down time.

The steps for this are documented as a known limitation: https://secrets-store-csi-driver.sigs.k8s.io/known-limitations.html#how-to-fetch-the-latest-content-with-release-v0014-and-earlier-or-without-auto-rotation-feature-enabled. You can enable the secret autorotation and restart the deployment after the kubernetes secret has been updated to include the new secret.

Secondly, is there any recommended way to restart the Pods when the secret template changes:

There is no recommended way. It is up to the application owner. We do have some suggestions based on different scenarios: https://secrets-store-csi-driver.sigs.k8s.io/topics/secret-auto-rotation.html#auto-rotation-of-mounted-contents-and-synced-kubernetes-secrets

[vivuu1989]

@aramase , not clear what exactly need to be performed. I used below command while installing the CSI driver in the kubernetes cluster. helm install --set secrets-store-csi-driver.syncSecret.enabled=true kv-csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure -n kube-system.

I think, we are discussing 2 differentbthings and i will explain you the exact scenario, where I am facing the issue.

In the above example, say, I added additionaly one more new secret "MyAPPA_SECRET4_NAME4" to the values.yaml file under "Keyvaults.Secrets" section. And the same name secret I created in the Keyvault

So inorder to get the secretproviderclass template updated with the new secet keyobjects, We decided to release the helm upgrade( which will perform the deployment, secret, secretproviderclass etc"). So in the helm upgrad --dry-run I could see the newly added key is coming properly inside the rendered secretproviderclass elements and Deployment environment elemens. Butr the helm release getting failed by stating that the newly added key is not present.

So iF i uninstall the helm release and re do the helm upgrade alone. its able to create those newly added the secrets. But with this approach i cant procedd this to Prod environment.

so each time when we update the secret in keystore and adding the same secret key under , we are fine to release the hlem upgrade. But it seems, helm upgrade still cant fetch the newly added secret but why helm upgrade will fail to update the secret during the upgrade. ? can you try to explain me here with my sceario.

[aramase]

So iF i uninstall the helm release and re do the helm upgrade alone. its able to create those newly added the secrets. But with this approach i cant procedd this to Prod environment.

@vivuu1989 I think we're talking about the same thing. If you want the new secret to be updated in the Kubernetes secret, (1) you can enable auto rotation, wait for the Kubernetes secret to be updated and then do a rolling update of your deployment or (2) delete the Kubernetes secret created by the CSI driver prior to helm upgrade, so the new Kubernetes secret will be created that includes MyAPPA_SECRET4_NAME4.

The synced Kubernetes secret is only updated as part of rotation or when it doesn't exist. The doc I shared (https://secrets-store-csi-driver.sigs.k8s.io/known-limitations.html#how-to-fetch-the-latest-content-with-release-v0014-and-earlier-or-without-auto-rotation-feature-enabled) has the exact steps that need to be done to get a new secret as env var.

[vivuu1989]

Thanks for the detailed explanation. So for the second approach, "delete the Kubernetes secret created by the CSI driver prior to helm upgrade, so the new Kubernetes secret will be created that includes MyAPPA_SECRET4_NAME4",

can I do dynamically identify the existing secret and delete it without causing any down time.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.