enable velero backups by using local instead of hostpath

[damoon]

by using local instead of hostpath it would be possible to use velero with restic for backups. velero with restic for volume backups can not support to backup host-path, but local is supported.

i use cassandra with rancher/local-path-provisioner for volumes to get bare metal disk performance, but backups are also nice...

velero limitations https://github.com/vmware-tanzu/velero/blob/master/site/docs/master/restic.md#limitations

host-path vs local https://kubernetes.io/docs/concepts/storage/volumes/#local https://kubernetes.io/docs/concepts/storage/volumes/#hostpath https://kubernetes.io/blog/2019/04/04/kubernetes-1.14-local-persistent-volumes-ga/

as far as a have read the code, the only place that would need to change is https://github.com/rancher/local-path-provisioner/blob/655eac7962bc1dbafb4bdec60b86bc8bc76b307c/provisioner.go#L212

does a technical reason exist not to use local? would it break current deployments to have a mixture of volumes types? is a merge request welcome?

[yesnomaybeyes]

I'm looking at this too, would LOVE to use Velero with local-path-provisioner, especially with k3s.

@damoon have you made the change mentioned and tried it with success? I'm going to try to do it this week but wanted to know if you saw any issues yourself.

[damoon]

i have not implemented or tested anything yet. i hope for some plausibility feedback from the developers before putting one more thing on my todo list.

for now i use https://github.com/maorfr/cain to backup cassandra.

[ariep]

In order to enable Velero backups, I implemented this change locally. It seems to work fine.

I didn't replace the use of hostPath in the "helper pods", I wasn't sure if that makes sense. It might be useful to replace as well, if only to not depend on the availability of hostPaths.

Are you interested at all in a pull request with my changes? It would need some further work on documentation to make it complete.

[yasker]

@damoon I remembered the reason we're not using Local volume at first is it requires an existing directory, while HostPath can create it automatically for you. But later we'd introduced the helper pod on creation to help on the directory creation anyway, so it shouldn't matter now.

@ariep PRs are welcome, drafts are fine too. You said you didn't change the usage of HostPath, so what part did you change?

[felixfriedrich]

I am very interested in being able to use Velero with the local-path-provisioner. Is there anything I can do or help with to make this happen?

[ariep]

PRs are welcome, drafts are fine too.

OK good, I opened a pull request with only the minimal changes I made thus far. It would require more work, at least in documentation.

You said you didn't change the usage of HostPath, so what part did you change?

I did change the PersistentVolume objects that are created, replacing HostPath by Local. I didn't change the temporary hostPaths in the helper pods though.

[yasker]

@ariep The code part of PR looks good to me. Thanks. Can someone else give the PR a try? Make sure it works for Velero at least.

[yasker]

We found an issue with local volume. It breaks the provisioner on RKE. Revert the change for now and keep the issue open.

[damoon]

how does the issue look like?

[yasker]

@damoon IIRC, the path cannot be found after created on the host (by the helper pod) while using RKE, so the Pod gets stuck on the pending state.

[BenTheElder]

Perhaps this can be optional?

the path cannot be found after created on the host (by the helper pod) while using RKE, so the Pod gets stuck on the pending state.

cannot be found by what, exactly?

[yasker]

@BenTheElder We will need to look into the issue more to find out why. Maybe we can make it optional.

So does KIND prefer local volume too?

[schmitch]

well the problem was probably hostPathType := v1.HostPathDirectoryOrCreate this line. i.e. the path was not created. so to support local volumes correctly somebody needs to enhance the mr to create the directory if it is missing.

[yasker]

@schmitch The directory creation part is already there since we're deploying the helper pod to create the directory. It seems the problem is RKE is containerized and might have issues with the local volumes. I will take a deep look later to see if it's RKE's issue.

[yasker]

Yes, confirmed it's an RKE issue.

See https://github.com/rancher/rancher/issues/22061#issuecomment-682110455

[jeanlucmongrain]

what if there is a daemonset that create the directory?

[acejam]

@yasker Any updates to this?

[acejam]

@yasker Bump for updates!

[benosman]

I would also like this functionality, and give a +1 for making it optional.

I am also looking to use Rancher / RKE, so I had a look into how to get kubernetes local volumes to create.

I posted a reply on the Rancher issue: https://github.com/rancher/rancher/issues/22061#issuecomment-806093507

To get local volumes working on an RKE cluster extra_binds must be set for kubelet. https://rancher.com/docs/rke/latest/en/config-options/services/services-extras/#extra-binds

Note: the following config, will not fix this provisioner unless the local volume support is readded.

services:
  kubelet:
    extra_binds:
     - "/opt/local-path-provisioner:/opt/local-path-provisioner"

The directory /opt/local-path-provisioner will likely need to be created on the host.

I don't know to make this as seamless a process on rancher as possible, but it would still be good if it could be made to work with some configuration tweaks.

[coutantal]

Any update on this ? I would love to use Velero with local-path-provisioner on k3s.

[barnybadzoo]

I am also interested in this feature. Any Updates would be great..

[dyvol777]

Need it too, up

[raelix]

Need it too, up

[devinnasar]

👍

Bumping this again.

[ThomasSteinbach]

Had the same trouble with velero. Like this too 👍

[kmjayadeep]

I made a fork with cherry picked commit from https://github.com/rancher/local-path-provisioner/pull/91 here: https://github.com/kmjayadeep/local-path-provisioner

Docker image is pushed to : https://hub.docker.com/repository/docker/kmjayadeep/local-path-provisioner kmjayadeep/local-path-provisioner:velero-support

I have tested velero backup using restic in my k3s setup and it seems to work fine. Those who really need this feature, feel free to use the above tag

[josepowera]

For anyone trying to move from hostPath to Local type provisioner... The easy way to change config in k3s is:

to set to kmjaydeep fork kubectl -n kube-system set image deployment/local-path-provisioner local-path-provisioner=kmjayadeep/local-path- provisioner:velero-support

to go back to original (k3s 1.23 version) kubectl -n kube-system set image deployment/local-path-provisioner local-path-provisioner=rancher/local-path-provisioner:v0.0.21

When set to "local type" you have to recreate volumes (they're not updated automatically).

[varac]

Here's another fork which uses HostPath for persistent local storage: https://open.greenhost.net/stackspin/local-path-provisioner/

• Helm chart repo: https://open.greenhost.net/api/v4/projects/44/packages/helm/stable
• Container images

[laverya]

The code change is still really remarkably small - would it be a plausible path forwards to allow configuring whether to use Local Volumes vs HostPaths as part of the config? That way RKE could keep using HostPath while allowing us Velero users to make backups.

[laverya]

If that is a path forwards that the maintainers would like (@yasker?) I'm happy to implement it!