datashim
datashim copied to clipboard
Published
20 hours ago •
datashim-io
datashim-io
S3's bucket cannot mount into a pod in k8s.
Hello, I tried datashim in the local k8s which has a master node and it works well. But when I apply the datashim in another kubernetes(1master+10nodes) with the same way, The pvc can create successfully but cannot mount into a nginx pod. This is the log I found in a csi-s3 pod:
I0506 01:42:52.653413 1 s3-driver.go:81] Driver: ch.ctrox.csi.s3-driver
I0506 01:42:52.653734 1 s3-driver.go:82] Version: v1.1.1
I0506 01:42:52.653992 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0506 01:42:52.654008 1 driver.go:93] Enabling volume access mode: SINGLE_NODE_WRITER
I0506 01:42:52.655539 1 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I0506 01:42:53.358925 1 utils.go:97] GRPC call: /csi.v1.Identity/GetPluginInfo
I0506 01:42:53.358995 1 utils.go:98] GRPC request: {}
I0506 01:42:53.363472 1 identityserver-default.go:32] Using default GetPluginInfo
I0506 01:42:53.363499 1 utils.go:103] GRPC response: {"name":"ch.ctrox.csi.s3-driver","vendor_version":"v1.1.1"}
I0506 01:42:53.449295 1 utils.go:97] GRPC call: /csi.v1.Identity/Probe
I0506 01:42:53.449356 1 utils.go:98] GRPC request: {}
I0506 01:42:53.449481 1 utils.go:103] GRPC response: {}
I0506 01:42:53.452289 1 utils.go:97] GRPC call: /csi.v1.Identity/GetPluginInfo
I0506 01:42:53.452343 1 utils.go:98] GRPC request: {}
I0506 01:42:53.452504 1 identityserver-default.go:32] Using default GetPluginInfo
I0506 01:42:53.452519 1 utils.go:103] GRPC response: {"name":"ch.ctrox.csi.s3-driver","vendor_version":"v1.1.1"}
I0506 01:42:53.454342 1 utils.go:97] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0506 01:42:53.454391 1 utils.go:98] GRPC request: {}
I0506 01:42:53.454635 1 identityserver-default.go:53] Using default capabilities
I0506 01:42:53.454656 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}}]}
I0506 01:42:53.457727 1 utils.go:97] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0506 01:42:53.457778 1 utils.go:98] GRPC request: {}
I0506 01:42:53.457839 1 controllerserver-default.go:62] Using default ControllerGetCapabilities
I0506 01:42:53.457850 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 01:42:53.985399 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetInfo
I0506 01:42:53.985492 1 utils.go:98] GRPC request: {}
I0506 01:42:53.985640 1 nodeserver-default.go:40] Using default NodeGetInfo
I0506 01:42:53.985663 1 utils.go:103] GRPC response: {"node_id":"js-yangzhou-7.172.18.0.95"}
I0506 02:00:34.904110 1 utils.go:97] GRPC call: /csi.v1.Controller/CreateVolume
I0506 02:00:34.904591 1 utils.go:98] GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead","parameters":{"mounter":"goofys"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
I0506 02:00:34.907130 1 controllerserver.go:72] Got a request to create volume pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead
I0506 02:00:34.926703 1 controllerserver.go:105] create volume pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead
I0506 02:00:34.926763 1 utils.go:103] GRPC response: {"volume":{"capacity_bytes":10000000000000,"volume_context":{"mounter":"goofys"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}}
I0506 02:51:37.884415 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:37.884653 1 utils.go:98] GRPC request: {}
I0506 02:51:37.885203 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:37.897327 1 utils.go:97] GRPC call: /csi.v1.Node/NodeStageVolume
I0506 02:51:37.897356 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
I0506 02:51:37.901213 1 utils.go:103] GRPC response: {}
I0506 02:51:37.907690 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:37.907744 1 utils.go:98] GRPC request: {}
I0506 02:51:37.907883 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:37.914580 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:37.914641 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
I0506 02:51:37.915453 1 nodeserver.go:75] target /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount
device
readonly false
volumeId pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead
attributes map[mounter:goofys storage.kubernetes.io/csiProvisionerIdentity:1683337373458-8081-ch.ctrox.csi.s3-driver]
mountflags []
I0506 02:51:37.915757 1 mounter_goofys.go:26] Mounting using goofys volume pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead
I0506 02:51:37.915780 1 mounter_goofys.go:53] Mounting using goofys!
I0506 02:51:37.916581 1 mounter.go:56] Mounting fuse with command: goofys and args: [--endpoint=http://minio-service:9000 --type-cache-ttl 1s --stat-cache-ttl 1s --dir-mode 0777 --file-mode 0777 --http-timeout 5m -o allow_other --profile=pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead test /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount]
E0506 02:51:39.066557 1 utils.go:101] GRPC error: stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:51:39.592184 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:39.592245 1 utils.go:98] GRPC request: {}
I0506 02:51:39.592380 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:39.599875 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:39.599939 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:51:39.600537 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:51:40.696793 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:40.696860 1 utils.go:98] GRPC request: {}
I0506 02:51:40.696956 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:40.702772 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:40.702844 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:51:40.703246 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:51:42.707661 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:42.707728 1 utils.go:98] GRPC request: {}
I0506 02:51:42.707837 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:42.715232 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:42.715278 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:51:42.715697 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:51:46.731186 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:46.731374 1 utils.go:98] GRPC request: {}
I0506 02:51:46.731704 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:46.738983 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:46.739047 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:51:46.739843 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:51:54.775237 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:51:54.775298 1 utils.go:98] GRPC request: {}
I0506 02:51:54.775471 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:51:54.780254 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:51:54.780330 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:51:54.780787 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:52:10.863501 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:52:10.865249 1 utils.go:98] GRPC request: {}
I0506 02:52:10.865393 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:52:10.872299 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:52:10.872365 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:52:10.873118 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:52:42.934041 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:52:42.934107 1 utils.go:98] GRPC request: {}
I0506 02:52:42.934211 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:52:42.941844 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:52:42.941897 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:52:42.942286 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:53:46.969583 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:53:46.969950 1 utils.go:98] GRPC request: {}
I0506 02:53:46.970224 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:53:46.975533 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:53:46.975585 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:53:46.976374 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:55:48.997553 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:55:48.997620 1 utils.go:98] GRPC request: {}
I0506 02:55:48.997733 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:55:49.006357 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:55:49.006543 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:55:49.007343 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:57:51.068016 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:57:51.068091 1 utils.go:98] GRPC request: {}
I0506 02:57:51.068226 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:57:51.074388 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:57:51.074464 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:57:51.075257 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 02:59:53.120890 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 02:59:53.120953 1 utils.go:98] GRPC request: {}
I0506 02:59:53.121059 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 02:59:53.128886 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 02:59:53.128971 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 02:59:53.129648 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 03:01:55.177258 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 03:01:55.177323 1 utils.go:98] GRPC request: {}
I0506 03:01:55.177481 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 03:01:55.183865 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 03:01:55.183916 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 03:01:55.184759 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 03:03:57.236970 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 03:03:57.237205 1 utils.go:98] GRPC request: {}
I0506 03:03:57.237404 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 03:03:57.245216 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 03:03:57.245274 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 03:03:57.245934 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 03:05:59.299269 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 03:05:59.300758 1 utils.go:98] GRPC request: {}
I0506 03:05:59.300887 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 03:05:59.308847 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 03:05:59.308915 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 03:05:59.309778 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 03:08:01.355651 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 03:08:01.355717 1 utils.go:98] GRPC request: {}
I0506 03:08:01.355819 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 03:08:01.364002 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 03:08:01.364073 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 03:08:01.364949 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
I0506 03:10:03.412736 1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0506 03:10:03.414149 1 utils.go:98] GRPC request: {}
I0506 03:10:03.414329 1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0506 03:10:03.420006 1 utils.go:97] GRPC call: /csi.v1.Node/NodePublishVolume
I0506 03:10:03.420044 1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/globalmount","target_path":"/var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"mounter":"goofys","storage.kubernetes.io/csiProvisionerIdentity":"1683337373458-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead"}
E0506 03:10:03.420593 1 utils.go:101] GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/pods/bc165e04-35be-4a27-a6b7-515c88d6e981/volumes/kubernetes.io~csi/pvc-6eb0f5d4-507f-4c7b-a296-fa6d7c52fead/mount: connection refused
What caused this? Thanks for help.
@Trainbow check if the namespace in which you are creating the datasets is being watched by Datashim. See this link for an explanation
@srikumar003 Thank you for your prompt reply, I tried to see the label of "monitor-pods-datasets" and it was set to "enabled."
kubectl get ns dlf --show-labels | grep monitor-pods-datasets
dlf Active 3d3h app.kubernetes.io/name=dlf,monitor-pods-datasets=enabled
But the pod still cannot mount the pvc, and the pod displays the following error information:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 24m (x466 over 22h) kubelet Unable to attach or mount volumes: unmounted volumes=[source], unattached volumes=[source default-token-w8hn4]: timed out waiting for the condition
Warning FailedMount 15m (x121 over 22h) kubelet Unable to attach or mount volumes: unmounted volumes=[source], unattached volumes=[default-token-w8hn4 source]: timed out waiting for the condition
Warning FailedMount 3s (x671 over 22h) kubelet MountVolume.SetUp failed for volume "pvc-89f0da6c-cd62-4d8c-97eb-00254fa77294" : rpc error: code = Internal desc = stat /var/lib/kubelet/pods/175e4c0f-78f5-4e16-8719-0031237b0829/volumes/kubernetes.io~csi/pvc-89f0da6c-cd62-4d8c-97eb-00254fa77294/mount: connection refused
@Trainbow
Could you verify that you have the correct endpoint for the object storage service ? Protocol (http:// vs 'https://'), Port, etc.
Since this error appears in the mounter rather than the actual connector, it could be related to #160. Can you check if the kubelet path that you used for installing Datashim is the same as that in your cluster ?
@srikumar003 I tried the same install steps in the local docker-desktop k8s, and it works, so I think the minio's endpoint is correct.
Do you mean that the datashim's plugin didn't install in the correct path in the k8s cluster?
I exec a pod which named "csi-s3-xxxxx", and found the globalmount path is empty.
It seems installed in the correct path?