Delete PVC/PV/Pod automatically on node deletion

[baurmatt]

Hey,

we're a happy user of local-path-provisioner. Thank you for maintaining it! :)

One thing we realized while testing our processes is, that it is unnecessary hard to delete the PVC/PV/Pod on node rotation. Currently we use the following steps to migrate a pod to another node:

1. kubectl drain --ignore-daemonsets --delete-local-data worker-node1
2. kubectl delete pvc data-testpod-0
3. kubectl delete pod testpod-0
4. kubectl delete node worker-node1

Inspired from what TopoLVM has to offer, we would love to see that local-path-provisioner deletes the PVC/Pod on node deletion automatically. This would allow for an easier node rotation workflow:

1. kubectl drain --ignore-daemonsets --delete-local-data worker-node1
2. kubectl delete node worker-node1 => local-path-provisioner automatically deletes the PVC "data-testpod-0" on "worker-node1" and the pod.

Thanks for considering! :)

Regards, Matthias

[yasker]

There are a few things I would try to understand:

1. Why node drain didn't delete the pod? The draining step should automatically delete the pod if it's a managed pod (e.g. deployment/statefulset).
2. If the pod is going to be migrated, but data is still on the local node, how can the pod resume working on another node?
3. Regardless, how does TopoLVM delete the PVC when kubectl delete node happens? That's interesting..

[baurmatt]

1. Draining the node does delete the pod from the node. But the pod gets reschedule automatically with the configured PVC. Because the pod has an affinity to the PVC, it stays in the status "Pending". Later, if you delete the PVC, it isn't recreating the PVC. This seems to only happen on pod creation.
2. This is the job of the application running in the pod :) We, for example, think about using this for a Percona XtraDB Cluster which would automatically sync the data from other nodes in the cluster.
3. From my understanding TopoLVM uses a node finalizer to remove the PVC before the node is really delete in k8s. -> https://github.com/topolvm/topolvm/blob/master/docs/design.md#architecture

[yasker]

Thanks @baurmatt . Yes, those make sense.

However Local Path Provisioner doesn't have a controller to monitor the deletion of the node. Also, adding a finalizer to the node object will have the side effect of preventing the node from deletion if something went wrong on the local path provisioner part. So I am hesitated to go with that approach.

[baurmatt]

However Local Path Provisioner doesn't have a controller to monitor the deletion of the node.

I'm not deeply into operator/Go development, so please forgive me if I'm missing something here. From my understanding, this is what I'm requesting here.

Also, adding a finalizer to the node object will have the side effect of preventing the node from deletion if something went wrong on the local path provisioner part.

That's true, but isn't this exactly the use case finalizers have been created for?

So I am hesitated to go with that approach.

Do you see a another way to handle this problem? :)