Cloudstack CSI Driver cannot mount pvc after attach volume on VMware

[tungntvn23]

trafficstars

ISSUE TYPE

• Bug Report

COMPONENT NAME

• Storage csi driver, Kubernetes Service

CLOUDSTACK VERSION

• 4.17.2.0

CONFIGURATION

• Advanced networking with vmware 7
• Kubernetes service enabled
• Cloudstack CSI driver deployed

OS / ENVIRONMENT

• CentOS 7

SUMMARY

• When create kubernetes persistent volume claim, cloudstack create volume and attach to node success. But when node find new disk and mount, error happen and cannot mount new disk.

STEPS TO REPRODUCE

• Install the CSI driver in the K8s cluster using the command : kubectl apply -f https://github.com/apalia/cloudstack-csi-driver/releases/latest/download/manifest.yaml
• Deploy this storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cloudstack-custom
provisioner: csi.cloudstack.apache.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false
parameters:
  csi.cloudstack.apache.org/disk-offering-id: <id of custom disk offering>

• Deploy this pvs and pod:

kind: PersistentVolumeClaim
metadata:
  name: example-pvc
spec:
  storageClassName: cloudstack-custom
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
    - name: example
      image: busybox
      volumeMounts:
        - mountPath: "/data"
          name: example-volume
      stdin: true
      stdinOnce: true
      tty: true
  volumes:
    - name: example-volume
      persistentVolumeClaim:
        claimName: example-pvc

EXPECTED RESULTS

• PVC create, attach and mount successfull

ACTUAL RESULTS

• PVC cannot mount on node. Cluster logs:

2023-05-11T14:53:12.66819279Z stderr F {"level":"warn","ts":1683816792.6673605,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host0/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668231033Z stderr F {"level":"warn","ts":1683816792.6674378,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host1/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668293633Z stderr F {"level":"warn","ts":1683816792.6674924,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host2/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668320007Z stderr F {"level":"warn","ts":1683816792.667541,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host3/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668342763Z stderr F {"level":"warn","ts":1683816792.6675792,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host4/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668357687Z stderr F {"level":"warn","ts":1683816792.667621,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host5/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668386373Z stderr F {"level":"warn","ts":1683816792.6677256,"caller":"mount/mount.go:121","msg":"Error running udevadm trigger executable file not found in $PATH\n","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}```

[weizhouapache]

@tungntvn23 what's the guest OS ?

we have noticed some difference between OSes, for example https://github.com/apache/cloudstack/issues/7316

[tungntvn23]

@weizhouapache I use default system vm template for k8s node Debian GNU/Linux 11 (bullseye) and centos 7 for management with vmware 7.

[tungntvn23]

i try execute command inside container and get error:

root@democluster04-node-18809e60f05:~# ctr --namespace=k8s.io task exec -t --exec-id bash1 6e17361f0d4e773da13ab3a32c24394a1fd4186198032d0c03a918b77afacb55 sh
# echo '- - -' > /sys/class/scsi_host/host0/scan
sh: can't create /sys/class/scsi_host/host0/scan: Read-only file system
# whoami
root
#

I also check privileged is enable.

[zap51]

Hi, @kiranchavala @weizhouapache An official CSI for CloudStack would be really good. I've had experience with https://github.com/apalia/cloudstack-csi-driver, looks no longer maintained and doesn't work like with detaching volumes from the nodes, working with projects etc., Please let us know if we have a roadmap for this, this would be really appreciated :)

[weizhouapache]

moved to 4.18.2.0 milestone