kopf on.delete issues and requirements

[sajuptpm]

Long story short

I have following issue with delete flow.

Description

Q1) Current behaviour: kubernetes deleteing the kind which is marked for deletion even when there is a Permanent error in on.delete handler

Expected behaviour: Kubernetes should not delete kind which is marked for deletion when there is a Permanent error in on.delete handler. We should able to trigger deletion again. - How can i put state of the CR object back to created/deployed state when there is an error/exception in on.delete handler ??

Q2) things which I tried:

• In "on.create" handler, I have added a finalizer using "kopf.storage.finalizers.block_deletion" method.
• Then "on.delete" handler, I am deleting the same finalizer using "kopf.storage.finalizers.allow_deletion" method after executing the deletion logic.

Issues: How to handle the following situations: - error/exception before invoking "kopf.storage.finalizers.allow_deletion" method (before deleting the finalizer which we added) ?? - How to recover from deletion error/exception without using "kubectl patch xxx -p '{"metadata":{"finalizers":[]}}' --type=merge" ?? - How to restore CR object when there is an error/exception in on.delete handler ??

Q3) I think, Kopf handler (on.delete) get executed only after marking the state of the CR object to deleting in etcd database, right ? - is there anyway to execute some custom logic before storing it in etcd database, for example validation logic ?

[nolar]

Generally, you cannot. It is not related to Kopf, it is how Kubernetes behaves: once a resource is marked for deletion (by setting its .metadata.deletionTimestamp), it cannot be un-marked. I have checked:

$ k3d cluster create

$ kubectl apply -f examples/crd.yaml
customresourcedefinition.apiextensions.k8s.io/kopfexamples.kopf.dev created

$ kubectl apply -f examples/obj.yaml
kopfexample.kopf.dev/kopf-example-1 created

Prevent actual deletion.

$ kubectl patch kex kopf-example-1 --type merge -p '{"metadata":{"finalizers":["x"]}}'
kopfexample.kopf.dev/kopf-example-1 patched

$ kubectl get -o yaml kex kopf-example-1
apiVersion: kopf.dev/v1
kind: KopfExample
metadata:
  creationTimestamp: "2021-01-11T21:47:59Z"
  finalizers:
  - x
  generation: 1

Mark for deletion.

$ kubectl delete kex kopf-example-1
kopfexample.kopf.dev "kopf-example-1" deleted
^C

$ kubectl get -o yaml kex kopf-example-1
apiVersion: kopf.dev/v1
kind: KopfExample
metadata:
  creationTimestamp: "2021-01-11T21:47:59Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2021-01-11T21:49:54Z"
  finalizers:
  - x
  generation: 2

Un-mark from deletion (does not work).

$ kubectl patch kex kopf-example-1 --type merge -p '{"metadata":{"deletionTimestamp":null}}'
kopfexample.kopf.dev/kopf-example-1 patched (no change)

$ kubectl get -o yaml kex kopf-example-1
apiVersion: kopf.dev/v1
kind: KopfExample
metadata:
  creationTimestamp: "2021-01-11T21:47:59Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2021-01-11T21:49:54Z"
  finalizers:
  - x
  generation: 2

By the way, Kopf implicitly adds its finalizer when it has something to do on deletion, i.e. when it has any @kopf.on.delete handlers — unless optional=True is passed to the decorator.

As such, there is no need to add your own finalizers — just relying on Kopf's one might be sufficient (depends on the use-case).

---

How to recover from deletion error without using "kubectl path xxx -p '{"metadata":{"finalizers":[]}}' --type=merge" ??

Sadly, there is no other way at the moment except as manually patch-fixing the resources if you introduce your own finalizers.

Using Kopf's own finalizer should be sufficient though. When a permanent error is raised, the deletion handler is considered as failed, but the processing ends, the resource is released and actually deleted. It looks like this (just checked):

[default/kopf-example-1] Handler 'delete_fn' failed permanently: boo
[default/kopf-example-1] Deletion is processed: 0 succeeded; 1 failed.
[default/kopf-example-1] Removing the finalizer, thus allowing the actual deletion.
[default/kopf-example-1] Patching with: {'metadata': {'finalizers': []}}

---

How to restore CR object when there is an error in on.delete handler ??

Perhaps, not raising permanent errors is the way. You can raise a temporary error, or just arbitrary errors and configure their interpretation as temporary/permanent with the errors=... option.

---

I think, Kopf handler (on.delete) get executed only after marking the state of the CR object to deleting in etcd database, right ?

Right. It is a philosophy of Kubernetes in general: having resources that describe the "desired" state of something, manage them via a pre-existing K8s API & CLI, and let the operators do the "reconciliation" (synchronisation) of the "actual" state to the "desired" state over time.

is there anyway to execute some custom logic before storing it in etcd database, for example validation logic ?

Currently, no, but it is planned for the admission hooks in the milestone 30 (which comes directly after the milestone 29, which is now in the release-candidate stage as 1.29.0rc1, with rc2 coming soon).

I can neither confirm nor deny that it will be possible to validate the deletion marks — I didn't play with that system yet.