volume mode "Ephemeral" not supported by driver secrets-store.csi.k8s.io (no CSIDriver object)

[Chakravarti-Baratam]

trafficstars

Hi,

I am trying to mount Vault Secrets through Container Storage Interface (CSI) Volume. https://learn.hashicorp.com/tutorials/vault/kubernetes-secret-store-driver

I am getting below error: MountVolume.NewMounter initialization failed for volume "secrets-store-inline" : volume mode "Ephemeral" not supported by driver secrets-store.csi.k8s.io (no CSIDriver object)

Can you please share your thoughts on this?

Thanks, Chakri.

[tomhjp]

It sounds like maybe the Secrets Store CSI Driver isn't installed? If you run kubectl get pods, do you see pods for both the driver and the Vault CSI provider? This is the section that installs the driver: https://learn.hashicorp.com/tutorials/vault/kubernetes-secret-store-driver#install-the-secrets-store-csi-driver

[Chakravarti-Baratam]

Secrets Store CSI Driver is installed. And I can see pods for both the drive and vault csi provider.

[ramanagali]

I have the same issue My Vault server is provisioned using vagrant running in my mac (using https://github.com/ramanagali/vault-server), set IP as 192.168.10.10, 8200 port forwarded My K8S cluster provisioned using vagrant running in my mac (using https://github.com/ramanagali/k8s-cluster), kube-api server IP as 192.168.56.10, 6443 port forwarded

• CSI Driver installed in K8S cluster, its pod running in default namespace helm upgrade -i csi secrets-store-csi-driver/secrets-store-csi-driver --set syncSecret.enabled=true

• In K8s Cluster created sa vault-auth, clusterrolebinding role-tokenreview-binding with system:auth-delegator role

• Rest all followed as per https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-secret-store-driver

• Vault CSI Provider installed, configured SecretProviderClass something like this

  helm install vault hashicorp/vault --set "server.enabled=false" --set "injector.enabled=false" --set "csi.enabled=true" using individual vault server

  apiVersion: secrets-store.csi.x-k8s.io/v1
  kind: SecretProviderClass
  metadata:
    name: vault-database
  spec:
    provider: vault
    parameters:
      vaultAddress: "http://192.168.10.10:8200"
      roleName: "database"
      objects: |
        - objectName: "db-password"
          secretPath: "secret/data/db-pass"
          secretKey: "password"
  

• when i run k get po

  NAME                                 READY   STATUS              RESTARTS   AGE
  csi-secrets-store-csi-driver-hhfnj   3/3     Running             0          27m
  multitool                            1/1     Running             0          15m
  vault-csi-provider-cqzf6             1/1     Running             0          26m
  webapp                               0/1     ContainerCreating   0          15m
  

• multitool pod installed in default ns, to check value is accessible from K8S. vault call are accessible

  k exec -it multitool -- sh                                                                                                                      ✔
  / #
  / # curl -H "X-Vault-Request: true" \
  >     -H "X-Vault-Token: hvs.DfEsZbi6K9HWzhiBPVIcRypG" \
  >     http://192.168.10.10:8200/v1/secret/db-pass
  {"request_id":"ff87da05-a8bd-6b77-a2f2-12de1d24d587","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"password":"db-secret-password"},"wrap_info":null,"warnings":null,"auth":null}
  / #
  / # exit
  

  in fact.. these calls are also 200 okay from multitool

  curl -H "X-Vault-Token: hvs.*******************" \
      -X LIST http://192.168.10.10:8200/v1/auth/kubernetes/role | jq
  
  curl -X POST \
      --data '{"role": "database","jwt": $K8S_TOKEN }' \
      http://192.168.10.10:8200/v1/auth/kubernetes/login
  

• the actual problem is my webapp pod is throwing error MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/webapp, err: rpc error: code = Unknown desc = error making mount request: couldn't read secret "db-password": Error making API request.

  k describe po webapp
  ....
  
  Events:
    Type     Reason       Age                    From               Message
    ----     ------       ----                   ----               -------
    Normal   Scheduled    26m                    default-scheduler  Successfully assigned default/webapp to worker1
    Warning  FailedMount  9m1s (x8 over 24m)     kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline kube-api-access-s8dd6]: timed out waiting for the condition
    Warning  FailedMount  4m27s (x2 over 6m43s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[kube-api-access-s8dd6 secrets-store-inline]: timed out waiting for the condition
    Warning  FailedMount  15s (x21 over 26m)     kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/webapp, err: rpc error: code = Unknown desc = error making mount request: couldn't read secret "db-password": Error making API request.
  
  URL: GET http://192.168.10.10:8200/v1/secret/data/db-pass
  Code: 404. Errors:
  
  

as i stated above curl -H "X-Vault-Request: true" -H "X-Vault-Token: hvs.DfEsZbi6K9HWzhiBPVIcRypG" http://192.168.10.10:8200/v1/secret/db-pass call working fine from multitool pod

could someone or @tomhjp help me, what's the issue here?

[tomhjp]

@Chakravarti-Baratam sorry for the delay. Could you share full repro steps please? Kubernetes version + provider (GKE, or minikube, or kind etc), and all the helm install commands + config you're running that will make this repro on a fresh cluster?

I've just re-run the tutorial you linked, and it still works for me, so I think there must be some deviations.

[tomhjp]

Going to close for now but do feel free to re-open if you'd like more help, or can provide some repro steps for an issue.

[smutkule]

facing the same issue, can anyone help here. :)

[tomhjp]

@smutkule please feel free to open a new issue with some detailed repro steps - as above, I was never able to reproduce this previously with the provided information.