ReadWriteMany mode fails to mount

[nadenf]

Given the following situation all within the same namespace.

NodeA --- PodA --- Volume A  ---+
                                | ---- PersistentVolumeClaim (RWM) ---- StorageClass
NodeB --- PodB --- Volume B  ---+

Volume A will successfully mount and you see this pod running:

juicefs-k3d-nodeA-pvc-467cfe22-0099-442f-8165-61be09295da1

But PodB will fail to start with this error:

Could not mount juicefs: rpc error: code = Internal desc = Could not mount juicef │
│ s: pods "juicefs-k3d-nodeB-pvc-467cfe22-0099-442f-8165-61be09295da1" not found  

And this error in the juicefs-csi-node for NodeB:

│ I1110 11:04:53.566859       1 juicefs.go:328] Mount: skip mounting for existing mount point "/jfs/pvc-467cfe22-0099-442f-8165-61be09295da1"                                                                                                                 │
│ I1110 11:04:53.566869       1 juicefs.go:331] Mount: add mount ref of configMap of volumeId "pvc-467cfe22-0099-442f-8165-61be09295da1"                                                                                                                      │
│ I1110 11:04:53.566880       1 client.go:65] Get pod juicefs-k3d-nodeB-pvc-467cfe22-0099-442f-8165-61be09295da1                                                                                                                                      │
│ I1110 11:04:53.571303       1 client.go:68] Can't get pod juicefs-k3d-nodeB-pvc-467cfe22-0099-442f-8165-61be09295da1 namespace system-filesystem: pods "juicefs-k3d-nodeB-pvc-467cfe22-0099-442f-8165-61be09295da1" not found               │
│ E1110 11:04:53.571324       1 driver.go:60] GRPC error: rpc error: code = Internal desc = Could not mount juicefs: rpc error: code = Internal desc = Could not mount juicefs: pods "juicefs-k3d-nodeB-pvc-467cfe22-0099-442f-8165-61be09295da1" not │
│ I1110 11:06:55.655175       1 node.go:217] NodeGetCapabilities: called with args                                                                                                                                                                            ```

[zwwhdls]

Hi @nadenf , can you tell which version did you use? And would you check if path "/jfs/pvc-467cfe22-0099-442f-8165-61be09295da1" is a mountpoint when the issue happened?

kubectl -n <namespace> exec -it juicefs-csi-node-<xxx> bash
df | grep /jfs/pvc-467cfe22-0099-442f-8165-61be09295da1

[nadenf]

Version:

juicedata/juicefs-csi-driver:v0.10.6

root@juicefs-csi-node-c7k49 (on NodeB):

JuiceFS:redis       1099511627776       128 1099511627648   1% /jfs/pvc-467cfe22-0099-442f-8165-61be09295da1

[zwwhdls]

Hi @nadenf ,

I can't reproduce this issue. Can you provide complete log of csi-node pod on NodeB?

[nadenf]

It's just a repeat of the logs above. Which component is responsible for launching PVC pods ?

juicefs-k3d-nodeB-pvc-467cfe22-0099-442f-8165-61be09295da1

[zwwhdls]

csi-node pod on NodeB is responsible for launching PVC pods. Can you search in juicefs-csi-node-c7k49 log whether there is a record of pod creation and deletion?

Also confirm whether csi is deployed in system-filesystem, and whether there is pod juicefs-k3d-nodeB-pvc-467cfe22-0099-442f-8165-61be09295da1 in system-filesystem.

[zwwhdls]

No feedback for a long time. closing, reopen when needed.