Failed to mount device in kubelet

[yujunz]

Normal   SuccessfulAttachVolume  63s                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-bbd05807-f1a9-11e9-9590-0800273e5475"
Warning  FailedMount             31s (x7 over 62s)    kubelet, minikube        MountVolume.MountDevice failed for volume "pvc-bbd05807-f1a9-11e9-9590-0800273e5475" : driver name csi.juicefs.com not found in the list of registered CSI drivers

[yujunz]

Possibly related to https://github.com/kubernetes/kubernetes/issues/68688

Logs from kubelet

Oct 18 13:17:29 minikube kubelet[4047]: E1018 13:17:29.600118    4047 plugin_watcher.go:116] error failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/csi.juicefs.com/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration when handling create event: "/var/lib/kubelet/plugins/csi.juicefs.com/csi.sock": CREATE
Oct 18 13:18:16 minikube kubelet[4047]: I1018 13:18:16.283896    4047 log.go:172] http: multiple response.WriteHeader calls
Oct 18 13:18:16 minikube kubelet[4047]: I1018 13:18:16.284065    4047 log.go:172] http: multiple response.WriteHeader calls
Oct 18 13:18:16 minikube kubelet[4047]: I1018 13:18:16.284153    4047 log.go:172] http: multiple response.WriteHeader calls
Oct 18 13:18:26 minikube kubelet[4047]: I1018 13:18:26.714731    4047 log.go:172] http: multiple response.WriteHeader calls
Oct 18 13:18:26 minikube kubelet[4047]: I1018 13:18:26.717948    4047 log.go:172] http: multiple response.WriteHeader calls
Oct 18 13:18:26 minikube kubelet[4047]: I1018 13:18:26.718131    4047 log.go:172] http: multiple response.WriteHeader calls
Oct 18 13:19:19 minikube kubelet[4047]: I1018 13:19:19.427165    4047 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-j9zn4" (UniqueName: "kubernetes.io/secret/bbd2dc4f-f1a9-11e9-9590-0800273e5475-default-token-j9zn4") pod "juicefs-app" (UID: "bbd2dc4f-f1a9-11e9-9590-0800273e5475")
Oct 18 13:19:19 minikube kubelet[4047]: I1018 13:19:19.427234    4047 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-bbd05807-f1a9-11e9-9590-0800273e5475" (UniqueName: "kubernetes.io/csi/csi.juicefs.com^pvc-bbd05807-f1a9-11e9-9590-0800273e5475") pod "juicefs-app" (UID: "bbd2dc4f-f1a9-11e9-9590-0800273e5475")
Oct 18 13:19:19 minikube kubelet[4047]: E1018 13:19:19.427440    4047 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/csi/csi.juicefs.com^pvc-bbd05807-f1a9-11e9-9590-0800273e5475\"" failed. No retries permitted until 2019-10-18 13:19:19.927409768 +0000 UTC m=+1064.841857054 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-bbd05807-f1a9-11e9-9590-0800273e5475\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^pvc-bbd05807-f1a9-11e9-9590-0800273e5475\") pod \"juicefs-app\" (UID: \"bbd2dc4f-f1a9-11e9-9590-0800273e5475\") "
Oct 18 13:19:19 minikube kubelet[4047]: I1018 13:19:19.932522    4047 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-bbd05807-f1a9-11e9-9590-0800273e5475" (UniqueName: "kubernetes.io/csi/csi.juicefs.com^pvc-bbd05807-f1a9-11e9-9590-0800273e5475") pod "juicefs-app" (UID: "bbd2dc4f-f1a9-11e9-9590-0800273e5475")
Oct 18 13:19:19 minikube kubelet[4047]: I1018 13:19:19.935901    4047 operation_generator.go:1205] Controller attach succeeded for volume "pvc-bbd05807-f1a9-11e9-9590-0800273e5475" (UniqueName: "kubernetes.io/csi/csi.juicefs.com^pvc-bbd05807-f1a9-11e9-9590-0800273e5475") pod "juicefs-app" (UID: "bbd2dc4f-f1a9-11e9-9590-0800273e5475") device path: "csi-2de48477a0909fce0d3a0a084559ce09024de563db2fec00919cfdec61787401"
Oct 18 13:19:20 minikube kubelet[4047]: I1018 13:19:20.034496    4047 operation_generator.go:498] MountVolume.WaitForAttach entering for volume "pvc-bbd05807-f1a9-11e9-9590-0800273e5475" (UniqueName: "kubernetes.io/csi/csi.juicefs.com^pvc-bbd05807-f1a9-11e9-9590-0800273e5475") pod "juicefs-app" (UID: "bbd2dc4f-f1a9-11e9-9590-0800273e5475") DevicePath "csi-2de48477a0909fce0d3a0a084559ce09024de563db2fec00919cfdec61787401"
Oct 18 13:19:20 minikube kubelet[4047]: I1018 13:19:20.042715    4047 operation_generator.go:507] MountVolume.WaitForAttach succeeded for volume "pvc-bbd05807-f1a9-11e9-9590-0800273e5475" (UniqueName: "kubernetes.io/csi/csi.juicefs.com^pvc-bbd05807-f1a9-11e9-9590-0800273e5475") pod "juicefs-app" (UID: "bbd2dc4f-f1a9-11e9-9590-0800273e5475") DevicePath "csi-2de48477a0909fce0d3a0a084559ce09024de563db2fec00919cfdec61787401"
Oct 18 13:19:20 minikube kubelet[4047]: E1018 13:19:20.047087    4047 csi_attacher.go:335] kubernetes.io/csi: attacher.MountDevice failed: driver name csi.juicefs.com not found in the list of registered CSI drivers

[chengyu-l]

这是插件注册有问题，是csi-driver的问题，这个问题可以解决

[zwwhdls]

JuiceFS CSI driver implements the NodeGetInfo call. closing, reopen when reproduced.