k8s-csi-s3
k8s-csi-s3 copied to clipboard
FailedMount
Hello!
Sometimes I have problem with mounting a bucket to pods.
From k8s I get FailedMount
with the message: Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-qjr9f data]: timed out waiting for the condition
After restart csi pods (csi-provisioner-s3-0, csi-attacher-s3-0, csi-s3-xxxxx) it works for any time and then crashes again with the FailedMount
error.
kubectl logs -n kube-system csi-s3-XXXXX csi-s3 output:
I0119 11:31:53.254133 1 mounter.go:122] Found matching pid 39 on path /var/lib/kubelet/pods/92f3b65c-8bf1-4514-ba2d-63b12b0c56d5/volumes/kubernetes.io~csi/data-bucket/mount
I0119 11:31:53.254178 1 mounter.go:87] Found fuse pid 39 of mount /var/lib/kubelet/pods/92f3b65c-8bf1-4514-ba2d-63b12b0c56d5/volumes/kubernetes.io~csi/data-bucket/mount, checking if it still runs
I0119 11:31:53.254226 1 mounter.go:147] Fuse process with PID 39 still active, waiting...
I0119 11:31:53.354755 1 mounter.go:147] Fuse process with PID 39 still active, waiting...
I0119 11:31:53.505635 1 mounter.go:147] Fuse process with PID 39 still active, waiting...
I0119 11:31:53.731230 1 mounter.go:147] Fuse process with PID 39 still active, waiting...
I0119 11:31:54.069091 1 mounter.go:147] Fuse process with PID 39 still active, waiting...
W0119 11:31:54.575966 1 mounter.go:138] Fuse process seems dead, returning
I0119 11:31:54.576087 1 nodeserver.go:137] s3: volume data has been unmounted.
I0119 11:31:58.582661 1 utils.go:97] GRPC call: /csi.v1.Node/NodeUnpublishVolume
E0119 11:31:58.586674 1 utils.go:101] GRPC error: rpc error: code = Internal desc = Unmount failed: exit status 32
Unmounting arguments: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount
Output: umount: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount: not mounted.
I0119 11:32:05.659607 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0119 11:32:05.675449 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0119 11:32:05.675714 1 nodeserver.go:100] target /var/lib/kubelet/pods/95568661-488f-4b8c-af1c-dbbc21d590dd/volumes/kubernetes.io~csi/data-bucket/mount
readonly false
volumeId data
attributes map[capacity:500Gi mounter:geesefs options:--list-type 2]
mountflags []
I0119 11:32:05.676084 1 mounter.go:63] Mounting fuse with command: geesefs and args: [--endpoint http://minio-host:9000 -o allow_other --log-file /dev/stderr --list-type 2 data: /var/lib/kubelet/pods/95568661-488f-4b8c-af1c-dbbc21d590dd/volumes/kubernetes.io~csi/data-bucket/mount]
2022/01/19 11:32:05.741157 main.INFO File system has been successfully mounted.
I0119 11:32:06.744437 1 nodeserver.go:117] s3: volume data successfully mounted to /var/lib/kubelet/pods/95568661-488f-4b8c-af1c-dbbc21d590dd/volumes/kubernetes.io~csi/data-bucket/mount
I0119 11:32:06.865094 1 utils.go:97] GRPC call: /csi.v1.Node/NodeUnpublishVolume
E0119 11:32:06.870086 1 utils.go:101] GRPC error: rpc error: code = Internal desc = Unmount failed: exit status 32
Unmounting arguments: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount
Output: umount: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount: not mounted.
I0119 11:32:07.470276 1 utils.go:97] GRPC call: /csi.v1.Node/NodeUnpublishVolume
E0119 11:32:07.474229 1 utils.go:101] GRPC error: rpc error: code = Internal desc = Unmount failed: exit status 32
Unmounting arguments: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount
Output: umount: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount: not mounted.
I0119 11:32:08.483579 1 utils.go:97] GRPC call: /csi.v1.Node/NodeUnpublishVolume
E0119 11:32:08.487524 1 utils.go:101] GRPC error: rpc error: code = Internal desc = Unmount failed: exit status 32
Unmounting arguments: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount
Output: umount: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount: not mounted.
I0119 11:32:10.504619 1 utils.go:97] GRPC call: /csi.v1.Node/NodeUnpublishVolume
E0119 11:32:10.507909 1 utils.go:101] GRPC error: rpc error: code = Internal desc = Unmount failed: exit status 32
Unmounting arguments: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount
Output: umount: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount: not mounted.