Finalizer for kubernetes resources created by a workflow

[boniek83]

Summary

I want to make sure that kubernetes resource created by a workflow is deleted when workflow fails or is deleted. Currently using onExit doesn't guarantee that. I can run kubectl delete workflow - it will skip onExit, I can run argo terminate (or press terminate in UI) - it will skip onExit. What happens if argo itself stops working?

Probably related to https://github.com/argoproj/argo-workflows/issues/7860

Use Cases

I want to make sure that SparkApplications (that use a lot of resources) started by a Workflow die with a Workflow.

---

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

[terrytangyuan]

Use resource template and specify ownerReference.

[boniek83]

@terrytangyuan This doesn't work with SparkApplications (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator). When setOwnerReference is set on resource of SparkApplication type SparkApplication stops working correctly.

[terrytangyuan]

Could you clarify what you meant by "stops working correctly"? That only sets an owner reference.

[boniek83]

@terrytangyuan It doesn't set it correctly: 4m33s Warning OwnerRefInvalidNamespace sparkapplication/test-exit-handlers-7pbqr-spark-pi ownerRef [argoproj.io/v1alpha1/Workflow, namespace: sparkjobs, name: test-exit-handlers-7pbqr, uid: 44e03b69-2243-4e7d-9872-4ac711df27b7] does not exist in namespace "sparkjobs" I have a situation in which workflows are created in another namespace than resource object themselves so automatic creation of owner references doesn't work. Can I somehow instruct Argo to set correct namespace in owner reference?

[Joibel]

Owner references cannot cross namespaces.

[boniek83]

So there is no support for cleaning up resources created by workflow in different namespaces than workflow itself?

[Joibel]

So there is no support for cleaning up resources created by workflow in different namespaces than workflow itself?

No integrated support, no.

Your use case is very much a corner case. You're:

• violating namespace boundaries (when you could probably avoid doing that, which would fix this).
• directly deleting Running workflows rather than ensuring they are completed in some form.

Whilst it would technically be possible to add support to fix this issue, it's not something I think is the right thing to do as it adds quite a lot of complexity, has workarounds, and has not had interest from others.

[boniek83]

Fair enough.