Talos v1.7.5+local-path-provisioner v0.0.28

[wingying]

Bug Report

Description

Hi,

I installed Talos v1.7.5, one controlpanel and one worker node, and I installed local-path-provisioner v0.0.28 as storageclass. ocal-path-provisioner v0.0.28 is installed successfully, however, when I tried to install any helmchart, like loki, git-server, pvc cannot mounted successfully. See error message below from loki pod, and the pod keep in creating status.

loki pod log:

Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    9s                default-scheduler  Successfully assigned cdp-foundation/loki-0 to worker-1
  Warning  FailedMount  3s (x5 over 10s)  kubelet            MountVolume.NewMounter initialization failed for volume "pvc-5c019138-a2d9-4b37-8032-ff71327dfcc7" : path "/var/local-path-provisioner/pvc-5c019138-a2d9-4b37-8032-ff71327dfcc7_cdp-foundation_storage-loki-0" does not exist

However, it display pv and pvc were created successfully from local-path-provisioner pod

local-path-provisioner pod log:

I0711 10:57:02.104276       1 controller.go:1346] provision "cdp-foundation/storage-loki-0" class "local-path": persistentvolume "pvc-5c019138-a2d9-4b37-8032-ff71327dfcc7" already exists, skipping
I0711 10:57:02.104552       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"cdp-foundation", Name:"storage-loki-0", UID:"5c019138-a2d9-4b37-8032-ff71327dfcc7", APIVersion:"v1", ResourceVersion:"368013", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-5c019138-a2d9-4b37-8032-ff71327dfcc7

More Information

Below are troubleshooting steps I did:

1. hostpath works, pod can bind pv and pvc successfully.
2. I tried different versions of local-path-provisioner, neither works.
3. I tried to install different helmchart of applications, use storageclass and set to local-path, all failure, and same behavior.
4. I change local-path config path followed your guide in https://www.talos.dev/v1.7/kubernetes-guides/configuration/local-storage/ switch to opt to var.
5. I even make a k8s job to set permission to give everyone read/write permission for /var, there is no use.

Could you help me the issue?