Allow disabling a Kustomization's finalizer

[julienduchesne]

Discussed here: https://cloud-native.slack.com/archives/CLAJ40HV3/p1653481382666929 but opening this for further discussion

The prune setting of Kustomizations has a two-fold behavior. It will delete resources which are removed from source, as is expected from a GitOps tool and it also deletes all resources managed by the Kustomization through its finalizer, when it is deleted.

We'd like to be able to keep pruning but disable the finalizer for the following reasons:

1. Prevent a cascading deletion in case someone deletes the Kustomizationresource from the source (git). Flux manages itself on our clusters and it can be unexpected for users that deleting or modifying a single resource will cascade to others
2. Allow easier management of Kustomizations. They currently need to be suspended to be interacted with. Our usecase: we have a "global" Kustomization that manages other Kustomizations, which are located in each namespace. The global Kustomization is set to NOT prune to avoid a cascade deletion of the whole cluster on its deletion. If we got rid of the finalizer, we could manage everything with prune. This would allow us to do operations like renaming all Kustomizations in the cluster without any manual intervention
3. Safeguard against accidental deletion. I understand that production clusters should have access restrictions in place. However, in dev for example, a user may wish to disable Flux and delete the Kustomization, not knowing that this will delete all of the namespace's resources

I'd propose a new Kustomization setting: finalizerMode with three modes: none (disabled), delabel (remove flux labels on managed resources, prune (current mode)

[makkes]

Please read https://fluxcd.io/docs/components/kustomize/kustomization/#garbage-collection.

[julienduchesne]

Please read https://fluxcd.io/docs/components/kustomize/kustomization/#garbage-collection.

Thanks, this seems to match my understanding of how GC works. Did I get anything wrong in my issue description?

[stefanprodan]

Having annotations as a bandaid for not setting proper RBAC is not something I would consider. If users don't understand how Flux works, a better way to protect the cluster from destructive actions, would be to remove write & delete actions in RBAC, so expect for cluster admins, no other user could edit or delete Flux objects.

[julienduchesne]

All of our manifests are generated from a monorepo. We have CODEOWNERs set up for the paths that generate Kustomizations, but even a code owner can make a mistake. We'd like to have Flux be a little less destructive, and we have no use for the finalizer

Also, point 2, this would allow us to do more with GitOps, less manual actions.

And I didn't think things through, I meant a Kustomization setting, not an annotation. Updated the description.

Sorry, I don't mean to be pushy. I just want to be sure I am fully understood before dropping this, if you still don't agree 🙂

[zxbyoyoyo]

I also encountered the same problem, when setting prune=true there are two behaviors,

1. deleting a single resource in Git will delete it from K8s;
2. when the Kustomize object is deleted, all managed resources will be deleted from K8s.

If I want to use the first behavior and don't need the second behavior, I need to add annotation kustomize.toolkit.fluxcd.io/prune: disabled to each resource to avoid not being recycled, which is very unfriendly , I also recommend providing a new field for the second behavior to support global control at the fluxcd level