Microk8s - MountVolume.SetUp failed

[fornof]

What steps did you take and what happened: [A clear and concise description of what the bug is, and what commands you ran.]

https://github.com/openebs/zfs-localpv/issues/235 this is about what I'm running into, but I didn't see how to solve it in the ticket:

   Warning  FailedMount  86s (x12 over 9m40s)  kubelet  MountVolume.SetUp failed for volume "pvc-3da01b7f-3eb3-427d-943d-42758527c8f2" : applyFSGroup failed for vol pvc-3da01b7f-3eb3-427d-943d-42758527c8f2: lstat /var/snap/microk8s/common/var/lib/kubelet/pods/b803ceff-d82e-4fe2-be34-bb05f6bee84d/volumes/kubernetes.io~csi/pvc-3da01b7f-3eb3-427d-943d-42758527c8f2/mount: no such file or directory                                               

I tried again using a deployment mount and I get:

persistent volume:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM
            STORAGECLASS           VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-2e49cd25-4b11-4618-817f-3095de775145   20Gi       RWO            Delete           Released   minio-operator/0-microk8s-ss-0-0   microk8s-hostpath      <unset>                          46d
pvc-3a97c089-e659-4d9e-91a6-62bae21f89f4   76Gi       RWO            Delete           Bound      postgres/postgres-fornof-zfs-pvc   openebs-zfspv-fornof   <unset>                          11h
pvc-3da01b7f-3eb3-427d-943d-42758527c8f2   76Gi       RWO            Delete           Bound      db/postgres-fornof-zfs-pvc         openebs-zfspv-fornof   <unset>                          29m
persistent volume claims : 

{ "apiVersion": "v1", "items": [ { "apiVersion": "v1", "kind": "PersistentVolumeClaim", "metadata": { "creationTimestamp": "2024-06-22T16:59:42Z", "finalizers": [ "kubernetes.io/pvc-protection" ], "labels": { "app.kubernetes.io/component": "primary", "app.kubernetes.io/instance": "postgres-bitnami", "app.kubernetes.io/name": "postgresql" }, "name": "data-postgres-bitnami-postgresql-0", "namespace": "db", "resourceVersion": "17057302", "uid": "9a5bb71b-76d0-4a91-a84b-78eb5f9790ee" }, "spec": { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "8Gi" } }, "volumeMode": "Filesystem" }, "status": { "phase": "Pending" } }, { "apiVersion": "v1", "kind": "PersistentVolumeClaim", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"postgres-fornof-zfs-pvc","namespace":"db"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"76Gi"}},"storageClassName":"openebs-zfspv-fornof"}}\n", "pv.kubernetes.io/bind-completed": "yes", "pv.kubernetes.io/bound-by-controller": "yes", "volume.beta.kubernetes.io/storage-provisioner": "zfs.csi.openebs.io", "volume.kubernetes.io/storage-provisioner": "zfs.csi.openebs.io" }, "creationTimestamp": "2024-06-22T17:14:01Z", "finalizers": [ "kubernetes.io/pvc-protection" ], "name": "postgres-fornof-zfs-pvc", "namespace": "db", "resourceVersion": "17059608", "uid": "3da01b7f-3eb3-427d-943d-42758527c8f2" }, "spec": { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "76Gi" } }, "storageClassName": "openebs-zfspv-fornof", "volumeMode": "Filesystem", "volumeName": "pvc-3da01b7f-3eb3-427d-943d-42758527c8f2" }, "status": { "accessModes": [ "ReadWriteOnce" ], "capacity": { "storage": "76Gi" }, "phase": "Bound" } } ], "kind": "List", "metadata": { "resourceVersion": "" } }

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE data-postgres-bitnami-postgresql-0 Pending 44m postgres-fornof-zfs-pvc Bound pvc-3da01b7f-3eb3-427d-943d-42758527c8f2 76Gi RWO openebs-zfspv-fornof 30m

storage classes: 

NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 77m openebs-zfspv-fornof zfs.csi.openebs.io Delete Immediate false 15h

openebs pods: 

NAME READY STATUS RESTARTS AGE openebs-localpv-provisioner-7cd9f85f8f-6d7cp 1/1 Running 0 76m openebs-zfs-localpv-controller-7fdcd7f65-74l55 5/5 Running 0 76m openebs-zfs-localpv-node-6dn8t 2/2 Running 0 76m

rob@media:/mnt$ zfs list
NAME                                                    USED  AVAIL     REFER  MOUNTPOINT
robin-makeup                                            299M  1.76T      283M  /mnt/robin-makeup-zfs
robin-makeup/pvc-3da01b7f-3eb3-427d-943d-42758527c8f2    24K  76.0G       24K  legacy
**What did you expect to happen:**
expected volume to mount

**The output of the following commands will help us better understand what's going on**:
(Pasting long output into a [GitHub gist](https://gist.github.com) or other [Pastebin](https://pastebin.com/) is fine.)

* `kubectl logs -f openebs-zfs-controller-f78f7467c-blr7q -n openebs -c openebs-zfs-plugin`
https://nopaste.net/B3aBBGpU1q
* `kubectl logs -f openebs-zfs-node-[xxxx] -n openebs -c openebs-zfs-plugin`
* https://nopaste.net/yjl29wBl0F
* `kubectl get pods -n openebs`

NAME READY STATUS RESTARTS AGE openebs-localpv-provisioner-7cd9f85f8f-6d7cp 1/1 Running 0 88m openebs-zfs-localpv-controller-7fdcd7f65-74l55 5/5 Running 0 88m openebs-zfs-localpv-node-6dn8t 2/2 Running 0 88m

* `kubectl get zv -A -o yaml`

kubectl get zv -A -o yaml apiVersion: v1 items:

• apiVersion: zfs.openebs.io/v1 kind: ZFSVolume metadata: creationTimestamp: "2024-06-22T17:14:01Z" finalizers:
  • zfs.openebs.io/finalizer generation: 2 labels: kubernetes.io/nodename: media name: pvc-3da01b7f-3eb3-427d-943d-42758527c8f2 namespace: openebs resourceVersion: "17059601" uid: 685593db-552a-4daa-838b-4b8137a30c1a spec: capacity: "81604378624" compression: "off" dedup: "off" fsType: zfs ownerNodeID: media poolName: robin-makeup recordsize: 128k volumeType: DATASET status: state: Ready kind: List metadata: resourceVersion: ""

**Anything else you would like to add:**
I'm trying to get postgresql up and going on kubernetes and having a hard time mounting the persistent volume. 
[Miscellaneous information that will assist in solving the issue.]
I'm using bitnami postgresql with persistence set to true and storage set to true as the same volume claim in the helm chart values. 

**Environment:**
- LocalPV-ZFS version
- openebs.io/version=4.0.0   

- Kubernetes version (use `kubectl version`):

Client Version: v1.30.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.1

- Kubernetes installer & version:
- MicroK8s v1.30.1 revision 6876

- Cloud provider or hardware configuration:
Ubuntu 2TB mirrored storage ZFS pool. 
- OS (e.g. from `/etc/os-release`):
```PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"