✨ allow preserving addons on delete

[alam0rt]

trafficstars

What type of PR is this?

feature

What this PR does / why we need it:

We are looking at migrating away from the managed addons (specifically kube-proxy) and wanting to use our own self-managed.

The issue is that both kube-system/daemonset/kube-proxys can't exist at the same time for obvious reason.

Removing the addon from .spec.addons will of course, delete the addon.

Deploying over the addon works, but again, removing the addon will remove the now self-managed addon.

The only way (without juggling multiple daemonsets, using taints/tolerations to run both at the same time, etc) for a clean migration is to (please let me know if a good way comes to mind):

• Deploy over the managed addon
• Remove the addon while preserving the resources on the cluster

We could potentially stop CAPA reconciliation and remove the addon manually, but kind of want to avoid that.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #

Special notes for your reviewer:

Checklist:

• [ ] squashed commits
• [ ] includes documentation
• [x] includes emoji in title
• [x] adds unit tests
• [ ] adds or updates e2e tests

Release note:

Allow preserving EKS addons.