rook icon indicating copy to clipboard operation
rook copied to clipboard

minio pvc FailedMount

Open clcc2019 opened this issue 2 years ago • 4 comments

MountVolume.MountDevice failed for volume "pvc-a27b1cfb-194e-4a1e-845f-342e0b792299" : rpc error: code = Internal desc = rbd image replicapool/csi-vol-6547ec07-a9a4-11ec-8794-4a4838419698 is still being used Warning FailedMount 3m20s kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[minio-token-bbp7n data]: timed out waiting for the condition Warning FailedMount 64s (x8 over 19m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data minio-token-bbp7n]: timed out waiting for the condition

use rbd unmap /dev/rbd*, restart pod

clcc2019 avatar Jul 15 '22 06:07 clcc2019

The error "is still being used" seems to indicate that the mount isn't allowed as long as another pod has it mounted. Did you run "rbd unmap" directly? The CSI volume plugin is expected to map/unmap it for you, or else it will not be able to track the mounting properly. Also see the CSI troubleshooting guide.

travisn avatar Jul 15 '22 15:07 travisn

RWO pod is expected to run on only one node at a time. if you have issue with node failure please try node-loss in the csi troubleshooting guide @travisn shared

Madhu-1 avatar Jul 19 '22 03:07 Madhu-1

RWO pod is expected to run on only one node at a time. if you have issue with node failure please try node-loss in the csi troubleshooting guide @travisn shared

This is not a duplicate mount issue, we store a lot of metadata in minio which makes the pod mount volume timeout

clcc2019 avatar Aug 08 '22 03:08 clcc2019

The ceph rbd is correctly mounted to the node and can access the file, but the pod shows that the mount has timed out. After about 12 hours, the pod is running.

clcc2019 avatar Aug 08 '22 03:08 clcc2019