Not working for subPath

[AmreeshTyagi]

Thanks for providing this provisioner. I am setting up MySQL HA using Kubernetes Stateful example.

I have tried this provisioner, but it is creating dynamic PV & mounting also but not in DEFAULT provided directory, but on this path /var/lib/kubelet/pods/82d24112-fc50-11e8-90e7-005056b146f6/volume-subpaths.

I found that it is happeing if I use subPath, otherwise it works well.

Any idea, how can I resolve this issue?

[steav]

Yes, same for me.

Working without subpath (findmnt): |-/vol1 datastore[/local-path-pvc/pvc-b499d69f-2252-11e9-8fc6-14dda9d41a28] zfs rw,relatime,xattr,noacl

Problem using subpath: |-/vol1 overlay[/mnt/datastore/local-path-pvc/pvc-c0de97a5-2254-11e9-8fc6-14dda9d41a28/subdir1] overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/

Creating the directory tree before deployment does not help.

[yasker]

@AmreeshTyagi @steav Can you elaborate the problem? Or provide a yaml for expected behavior? Does the container work?

[khassel]

same for me. Seems that the volume-mount does not work with a subpath. The pod.yaml example works as expected, but if you add a subpath it doesn't work. Executing kubectl exec volume-test -- sh -c "echo local-path-test > /data/test" will write a test-file somewhere in /var/lib/kubelet/pods/... but not in the directory provided by the pvc local-path-pvc

[yasker]

@khassel When you said add a subpath, it's like this? https://github.com/rancher/local-path-provisioner/blob/master/examples/pod-with-subpath.yaml

[khassel]

yes, exactly.

[DrMurx]

I came across the same issue:

1. Create a single node Rancher cluster
2. Install local-path-provisioner storage class
3. Enable cluster level monitoring built into RKE, and enable persistent storage for Prometheus on the local-path storage class

[khassel]

@yasker Could you reproduce this subPath issue? Can we expect a solution?

[yasker]

@khassel You're right, I am able to reproduce the issue now.

It seems due to https://github.com/kubernetes/kubernetes/issues/61456 , which is a problem with containerized kubelet. If you're using RKE, we have an issue tracking it with Rancher: https://github.com/rancher/rancher/issues/14836

I've also tried using LocalVolume instead of HostPath, but it doesn't work for RKE as well. Seems not much I can do at the moment except waiting for the RKE fix.

[khassel]

@yasker Thanks for this info.

[AntonSmolkov]

Hi! Workaround:

kubectl patch statefulset prometheus-cluster-monitoring --type='json' --namespace=cattle-prometheus -p '
- op: remove
  path: "/spec/template/spec/containers/0/volumeMounts/1/subPath"
'

Source: https://vividcode.io/prometheus-operator-persistence-with-rancher-local-path-provisioner/

Update: For all prometheuses, system-level and project-level, in all namespaces:

proms=$(kubectl get statefulsets --all-namespaces -o json |  jq -r '.items[] | select(.metadata.name | contains("prometheus-")) | [.metadata.name, .metadata.namespace] | join(";")')   

while IFS= read -r prom; do
kubectl patch statefulset $(echo $prom | cut -d ';' -f1) --type='json' --namespace=$(echo $prom | cut -d ';' -f2) -p '
- op: remove
  path: "/spec/template/spec/containers/0/volumeMounts/1/subPath"
'
done <<< "$proms"

[disaster37]

Hi, I have tested to patch as explain it. But after patch it, kubernetes redeploy statefullset and so overwrite the patched, ... So it doesn't work.

Have you any idea ?

[attrib]

@yasker https://github.com/kubernetes/kubernetes/issues/61456 and https://github.com/rancher/rancher/issues/14836 got closed, but from my understanding there it seems to be not really fixed? Or is this helpful and this issue could be fixed in the future?

[immanuelfodor]

FYI, we've spend half a day to debug why our Prometheus is filling up the /var/lib/docker folder when provisioned by Rancher Monitoring V2.5, and found this issue and also https://github.com/rancher/rancher/issues/29149

# prometheus volume set by the operator:

     - mountPath: /prometheus                                                                                                                                                                                                              
       name: prometheus-rancher-monitoring-prometheus-db                                                                                                                                                                                   
       subPath: prometheus-db

An update of the Prometheus Operator should help in this case, the fix was validated on the linked ticket 23 days ago.

[patrikbeno]

my findings:

- mountPath: /mnt/full
  name: local
- mountPath: /mnt/subpath
  name: local
  subPath: mysubpath


pod:~$ touch /mnt/full/in-volume-123
pod:~$ touch /mnt/subpath/in-subpath-123


## sts deployed, pod running

node:~$ sudo find / -name in-volume-123 -or -name in-subpath-123
/var/lib/docker/overlay2/712369d63e3a2e7d0460f4eee887fc154baf10156d006e0dd2074942a1c39cb4/diff/var/lib/local-path-provisioner/pvc-81f02487-b5c7-4853-afe0-2a498a7a56f4_ci_local-ci-gims-gims-experimental-ci-new-0/mysubpath/in-subpath-123
/var/lib/docker/overlay2/712369d63e3a2e7d0460f4eee887fc154baf10156d006e0dd2074942a1c39cb4/merged/var/lib/local-path-provisioner/pvc-81f02487-b5c7-4853-afe0-2a498a7a56f4_ci_local-ci-gims-gims-experimental-ci-new-0/mysubpath/in-subpath-123
/var/lib/kubelet/pods/46d3932d-27f0-4884-9655-78ed8b52cc0f/volume-subpaths/pvc-81f02487-b5c7-4853-afe0-2a498a7a56f4/ci-gims-gims-experimental-ci-new/8/in-subpath-123
/var/lib/local-path-provisioner/pvc-81f02487-b5c7-4853-afe0-2a498a7a56f4_ci_local-ci-gims-gims-experimental-ci-new-0/in-volume-123

## kubectl scale --replicas=0 sts/name

node:~$ sudo find / -name in-volume-123 -or -name in-subpath-123
/var/lib/docker/overlay2/712369d63e3a2e7d0460f4eee887fc154baf10156d006e0dd2074942a1c39cb4/diff/var/lib/local-path-provisioner/pvc-81f02487-b5c7-4853-afe0-2a498a7a56f4_ci_local-ci-gims-gims-experimental-ci-new-0/mysubpath/in-subpath-123
/var/lib/docker/overlay2/712369d63e3a2e7d0460f4eee887fc154baf10156d006e0dd2074942a1c39cb4/merged/var/lib/local-path-provisioner/pvc-81f02487-b5c7-4853-afe0-2a498a7a56f4_ci_local-ci-gims-gims-experimental-ci-new-0/mysubpath/in-subpath-123
/var/lib/local-path-provisioner/pvc-81f02487-b5c7-4853-afe0-2a498a7a56f4_ci_local-ci-gims-gims-experimental-ci-new-0/in-volume-123


## kubectl delete sts/name 

node:~$ sudo find / -name in-volume-123 -or -name in-subpath-123
/var/lib/docker/overlay2/712369d63e3a2e7d0460f4eee887fc154baf10156d006e0dd2074942a1c39cb4/diff/var/lib/local-path-provisioner/pvc-81f02487-b5c7-4853-afe0-2a498a7a56f4_ci_local-ci-gims-gims-experimental-ci-new-0/mysubpath/in-subpath-123
/var/lib/docker/overlay2/712369d63e3a2e7d0460f4eee887fc154baf10156d006e0dd2074942a1c39cb4/merged/var/lib/local-path-provisioner/pvc-81f02487-b5c7-4853-afe0-2a498a7a56f4_ci_local-ci-gims-gims-experimental-ci-new-0/mysubpath/in-subpath-123
/var/lib/local-path-provisioner/pvc-81f02487-b5c7-4853-afe0-2a498a7a56f4_ci_local-ci-gims-gims-experimental-ci-new-0/in-volume-123

subPath data seems to be persisted but outside the volume, which is obviously a bug.

v0.0.22

[jhoblitt]

I also just ran into this unwittingly in v0.0.22. If there is warning in the docs about subpath, I didn't find it.

If this isn't going to be fixed, how about using a webhook to reject subPath?

[IlyaSemenov]

I switched to https://github.com/openebs/dynamic-localpv-provisioner which supports subPath just fine.