Andy Zhang

@MysticalMount what's the `kubectl get clusterrole nfs-external-provisioner-role -o yaml`, the `system:serviceaccount:csinfs:csi-nfs-controller-sa` should already have following permissions: https://github.com/kubernetes-csi/csi-driver-nfs/blob/d9f6033fe9b052a2673765299861aaa845d57b50/charts/v4.8.0/csi-driver-nfs/templates/rbac-csi-nfs.yaml#L38-L40

can you check why user "system:serviceaccount:csinfs:csi-nfs-controller-sa" cannot list resource "volumesnapshotcontents" in API group "snapshot.storage.k8s.io" at the cluster scope? e.g. check service account `csinfs:csi-nfs-controller-sa` info, that's not related to CRD. and...

try reinstall the nfs driver if you have upgraded the csi driver, similar to https://github.com/kubernetes-csi/external-snapshotter/issues/975#issuecomment-1850291353

one fix is that adding a new flag, e.g. `--set .controller.enableSnapshotter=false` in case you totally don't need snapshot function.

fixed by https://github.com/kubernetes-sigs/azuredisk-csi-driver/pull/2521

> azure_common_linux.go:121] /dev/disk/azure is not populated, now try to parse 1:0:0:0 directly from the error msg, it seems you are missing such udev rule on the node, follow guide here...

here is the data disk parsing logic, the reliable way is creating that udev rule on the node, that would create device links under `/dev/disk/azure` https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/99a117cfe0ff682d364215f6e6ab9e056a5b6339/pkg/azuredisk/azure_common_linux.go#L120-L194

can you try creating the udev rule on one node and check whether it works? @xiaoping8385 if it's working, then I could provide you a daemonset to creating udev rule...

@xiaoping8385 just create that script, and it would work automatically. and then reschedule pod with disk volume onto that node.

@xiaoping8385 can you run following command on the node, and try again? This command will reload the udev rules and trigger the creation of any new device nodes that match...