FailedMount

[entasadar]

Hello! Sometimes I have problem with mounting a bucket to pods. From k8s I get FailedMount with the message: Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-qjr9f data]: timed out waiting for the condition

After restart csi pods (csi-provisioner-s3-0, csi-attacher-s3-0, csi-s3-xxxxx) it works for any time and then crashes again with the FailedMount error.

kubectl logs -n kube-system csi-s3-XXXXX csi-s3 output:

I0119 11:31:53.254133       1 mounter.go:122] Found matching pid 39 on path /var/lib/kubelet/pods/92f3b65c-8bf1-4514-ba2d-63b12b0c56d5/volumes/kubernetes.io~csi/data-bucket/mount
I0119 11:31:53.254178       1 mounter.go:87] Found fuse pid 39 of mount /var/lib/kubelet/pods/92f3b65c-8bf1-4514-ba2d-63b12b0c56d5/volumes/kubernetes.io~csi/data-bucket/mount, checking if it still runs
I0119 11:31:53.254226       1 mounter.go:147] Fuse process with PID 39 still active, waiting...
I0119 11:31:53.354755       1 mounter.go:147] Fuse process with PID 39 still active, waiting...
I0119 11:31:53.505635       1 mounter.go:147] Fuse process with PID 39 still active, waiting...
I0119 11:31:53.731230       1 mounter.go:147] Fuse process with PID 39 still active, waiting...
I0119 11:31:54.069091       1 mounter.go:147] Fuse process with PID 39 still active, waiting...
W0119 11:31:54.575966       1 mounter.go:138] Fuse process seems dead, returning
I0119 11:31:54.576087       1 nodeserver.go:137] s3: volume data has been unmounted.
I0119 11:31:58.582661       1 utils.go:97] GRPC call: /csi.v1.Node/NodeUnpublishVolume
E0119 11:31:58.586674       1 utils.go:101] GRPC error: rpc error: code = Internal desc = Unmount failed: exit status 32
Unmounting arguments: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount
Output: umount: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount: not mounted.

I0119 11:32:05.659607       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0119 11:32:05.675449       1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0119 11:32:05.675714       1 nodeserver.go:100] target /var/lib/kubelet/pods/95568661-488f-4b8c-af1c-dbbc21d590dd/volumes/kubernetes.io~csi/data-bucket/mount
readonly false
volumeId data
attributes map[capacity:500Gi mounter:geesefs options:--list-type 2]
mountflags []
I0119 11:32:05.676084       1 mounter.go:63] Mounting fuse with command: geesefs and args: [--endpoint http://minio-host:9000 -o allow_other --log-file /dev/stderr --list-type 2 data: /var/lib/kubelet/pods/95568661-488f-4b8c-af1c-dbbc21d590dd/volumes/kubernetes.io~csi/data-bucket/mount]
2022/01/19 11:32:05.741157 main.INFO File system has been successfully mounted.
I0119 11:32:06.744437       1 nodeserver.go:117] s3: volume data successfully mounted to /var/lib/kubelet/pods/95568661-488f-4b8c-af1c-dbbc21d590dd/volumes/kubernetes.io~csi/data-bucket/mount
I0119 11:32:06.865094       1 utils.go:97] GRPC call: /csi.v1.Node/NodeUnpublishVolume
E0119 11:32:06.870086       1 utils.go:101] GRPC error: rpc error: code = Internal desc = Unmount failed: exit status 32
Unmounting arguments: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount
Output: umount: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount: not mounted.

I0119 11:32:07.470276       1 utils.go:97] GRPC call: /csi.v1.Node/NodeUnpublishVolume
E0119 11:32:07.474229       1 utils.go:101] GRPC error: rpc error: code = Internal desc = Unmount failed: exit status 32
Unmounting arguments: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount
Output: umount: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount: not mounted.

I0119 11:32:08.483579       1 utils.go:97] GRPC call: /csi.v1.Node/NodeUnpublishVolume
E0119 11:32:08.487524       1 utils.go:101] GRPC error: rpc error: code = Internal desc = Unmount failed: exit status 32
Unmounting arguments: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount
Output: umount: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount: not mounted.

I0119 11:32:10.504619       1 utils.go:97] GRPC call: /csi.v1.Node/NodeUnpublishVolume
E0119 11:32:10.507909       1 utils.go:101] GRPC error: rpc error: code = Internal desc = Unmount failed: exit status 32
Unmounting arguments: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount
Output: umount: /var/lib/kubelet/pods/d496285b-ca1e-4082-a849-77336431ae82/volumes/kubernetes.io~csi/data-bucket/mount: not mounted.