Using Kubeflow Notebooks with CSI Secrets Store and PodDefault

[kdubovikov]

PodDefault with CSI Secrets Store volume counts seem not to be working when using Kubeflow Notebooks. The exact same issue is being described here: https://github.com/kubeflow/kubeflow/issues/6539

You get an empty directory when trying to mount a secret volume using a PodDefault like this:

apiVersion: "kubeflow.org/v1alpha1"
kind: PodDefault
metadata:
  name: aws-secret
  namespace: cd-msr
spec:
  desc: "aws-secret"
  selector:
    matchLabels:
      aws-secret: "true"
  serviceAccountName: default-editor
  volumeMounts:
  - name: secrets-store-inline
    mountPath: "/mnt/secrets"
    readOnly: true
  volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: secrets-provider

Reproduced on Kubeflow 1.5.1

[surajkota]

Hi @kdubovikov, thanks for reporting the issue, will take a look at this

[surajkota]

Hi @kdubovikov, I am unable to reproduce this. Here are he steps I followed:

1. Install secrets-store-csi-driver
2. Install secrets-store-csi-driver-provider-aws
3. Create a secret in AWS Secrets Manager named surajkota-s3-secret
4. Create a profile named kubeflow-user-example-com in Kubeflow
5. Add arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess and arn:aws:iam::aws:policy/SecretsManagerReadWrite permissions to the default-editor service account

export CLUSTER_NAME=
export CLUSTER_REGION=
eksctl create iamserviceaccount  --name default-editor --namespace kubeflow-user-example-com --attach-policy-arn  arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess --attach-policy-arn arn:aws:iam::aws:policy/SecretsManagerReadWrite --override-existing-serviceaccounts   --approve --cluster $CLUSTER_NAME --region $CLUSTER_REGION

6. Create a SecretProviderClass in the profile namespace with following configuration

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: s3-secret
  namespace: kubeflow-user-example-com
spec:
  parameters:
    objects: |
      - objectName: surajkota-s3-secret
        objectType: secretsmanager
        jmesPath:
        - objectAlias: access
          path: accesskey
        - objectAlias: secret
          path: secretkey
  provider: aws

7. Create a PodDefault with the following configuration

apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
  name: sync-s3-secret
  namespace: kubeflow-user-example-com
spec:
  desc: Sync S3 secret
  selector:
    matchLabels:
      sync-s3-secret: "true"
  volumes:
    - name: s3secret
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "s3-secret"
  volumeMounts:
    - mountPath: /mnt/secret-store
      name: s3secret
      readOnly: true

8. Create a notebook and select the Sync S3 secret configuration
9. Open the notebook and check the contents of /mnt/secret-store directory. I found the secret contents and it was not empty

!ls /mnt/secret-store
access	secret	surajkota-s3-secret

Can you try the above steps and let us know if these work?

[kdubovikov]

@surajkota, thanks for the quick and detailed response. We have validated that this works on Kubeflow 1.6.1 and EKS 1.23. This can be an issue for Kubeflow 1.5.1 and EKS 1.21 though. Same steps do not work in this setup

[surajkota]

Thanks for validating. Secrets store CSI driver and pod defaults have not changed recently. Feel free to reopen the issue if anyone is facing this issue