cluster-api-provider-oci

cluster-api-provider-oci copied to clipboard

The CSIDriver for blockvolume.csi.oraclecloud.com is missing from the oci-addons flavor. Only the fss.csi.oraclecloud.com CSIDriver is present.

See drivers in template: https://github.com/oracle/cluster-api-provider-oci/blob/main/templates/cluster-template-oci-addons.yaml#L539

Missing driver should look something like this:

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: blockvolume.csi.oraclecloud.com
spec:
  fsGroupPolicy: File

Users of this template will have difficulty using PVs properly.

Additionally, there is no default storage class in the addons template. Users may need to annotate a storage class themselves.

The cluster role for CCM is also not quite right. list and watch verbs are missing for configmap

@anders-swanson we have used the latest manifests from https://github.com/oracle/oci-cloud-controller-manager/releases/tag/v1.24.0 and CSIDriver mentioned above is not present. The CCM cluster role is also copied from there. Can you please check there to make sure I am right? With the oci addons latest template, I was able to create a block volume using the following manifests. Can you please explain what is the problem you are facing. My thinking is that we will have to ask the CCM team to fix it if you are having problems. Manifests below

$ cat pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mynginxclaim
spec:
  storageClassName: "oci-bv-encrypted"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
$

$ cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:latest
      ports:
        - name: http
          containerPort: 80
      volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: mynginxclaim

We can make the encrypted storage class as default.

The specific issue I noticed with storage classes is that volume mounts did not have the proper permissions. If a pod was running as nonroot (say uid 1000, gid 999), and fsGroup for the pod was 999, volume mounts would still be owned by root. CSIDriver is supposed to handle chown -R of volume mounts if fsGroup is specified: I did not see this happening.

Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod

I see the same issues in the linked repository:

1. Additional CSIDriver is missing
2. CCM role is fragmented
3. No default storage class (this may not be an issue)

@anders-swanson I asked the same question to CSI/CCM team, here is the answer I received. Answers begin ---- CSI Driver for block volume is not required and hence it is not present. CSI driver is required only if we want to modify the defaults. customers can create it if they want to change fsGroupPolicy. this is mentioned on the CSI readme troubleshooting section. So this looks like something which can be added cy customer if required.

Regarding CCM - Need more info as we have tested the releases, all required RBAC is present.

Answers end ----

The same defaults exisyts in OKE as well from what I understand. Can you please open tickets agains the CCM repo and get answers from there or point me to the tickets already oepend?

Yeah, I can do that. Will also provide a more detailed reproduction.

@anders-swanson CCM team has added the CSI driver to ther latest manifests, we have added the same to the template in main branch, it will be released shortly.

I'm facing the same problem trying to attach a volume on OKE. What is the fix for this? OKE: v1.29.1

attachdetach-controller  AttachVolume.Attach failed for volume "csi-82fb12c9-a9b8-49f2-914c-29db56c95af0" : CSINode 10.X.X.X does not contain driver blockvolume.csi.oraclecloud.com

AttachVolume.Attach failed for volume "csi-5c8ab8de-924d-4a5e-8d93-8229b57049d8" : timed out waiting for external-attacher of blockvolume.csi.oraclecloud.com CSI driver to attach volume ocid1.volume.oc1.eu-frankfurt-1.xxxxxxx

I'm facing the same problem trying to attach a volume on OKE. What is the fix for this? OKE: v1.29.1

attachdetach-controller  AttachVolume.Attach failed for volume "csi-82fb12c9-a9b8-49f2-914c-29db56c95af0" : CSINode 10.X.X.X does not contain driver blockvolume.csi.oraclecloud.com

AttachVolume.Attach failed for volume "csi-5c8ab8de-924d-4a5e-8d93-8229b57049d8" : timed out waiting for external-attacher of blockvolume.csi.oraclecloud.com CSI driver to attach volume ocid1.volume.oc1.eu-frankfurt-1.xxxxxxx

Having the same issue, and unsure if I need to deploy the CSI Volume via https://oracle.github.io/cluster-api-provider-oci/gs/install-csi.html

I have tried that and it errors on not being able to deploy to the control plane. https://github.com/oracle/oci-cloud-controller-manager/issues/479