Provide a way for kapp-controller to not take ownership of existing resources

[jorgemoralespou]

[Describe the problem/challenge you have When deploying a package, if there's a resource defined in the package contents that already exist in the cluster, kapp will take ownership of this resource. From that moment, if the package is removed, then the existing resource will be removed as well.

Describe the solution you'd like I would want to have an annotation on the package that defines whether existing resources should be absorbed as owned or not, so that if the annotation is present and the resource exists, kapp-controller reconciliation will fail indicating that the resource already exists and that can not be claimed as owned by the package. This will prevent taking incorrectly ownership of existing namespaces, and similar problems.

Anything else you would like to add: This is a conversation that unveiled this problem: https://kubernetes.slack.com/archives/C02GY94A8KT/p1645018465370589

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[benmoss]

Hmm, maybe this should be

deploy: {kapp: {adopt-resources: false}}

?

I think this requires a feature to be implemented in https://github.com/vmware-tanzu/carvel-kapp as well

[joe-kimmel-vmw]

kapp won't stomp an existing ownership, so if you have a configmap foo and you also have a package that will install that same configmap, but you don't want to change the ownership, you could block the whole thing by setting the label kapp.k14s.io/app: "my-ownership" . Current behavior is that then the package would fail to reconcile. Could this be solved by a "rebase rule" style ownership decoration that would say "copy the ownership from the existing record" ?

Also, seems like this would go with PackageInstall rather than the package itself, and there's some challenges around knowledge of the resources - it seems odd to me that you would already have the exact resource that a package is creating. Can you detail a specific case?

[jorgemoralespou]

Use case is that we have seen that most of the Packages define a namespace and that the namespace is configurable. There's a big problem when a user in a non malicious way sets that namespace to an existing namespace (e.g. kube-system). From that moment, as kube-system did already exist as it came with the k8s cluster, the package owns the namespace. If that package is then removed, kube-system namespace will also be removed, and with it, everything contained in that namespace, leaving the cluster in a non-recoverable state. There needs to be a way to prevent packages from "claiming ownership" of existing resources (namespaces being the most critical). And I don't think this should be at the PackageInstall level, as PackageInstall are user provided resources, and we shouldn't need to leave this up to the end user, but rather to the Package author or cluster manager.

[joe-kimmel-vmw]

thanks @jorgemoralespou - the example you give does sound pretty inconvenient 😆 . I just want to make sure I understand though - if the namespace is configurable, who is the person who chooses that config value? Isn't that also the right person to know whether it should be uninstalled along with everything else?

[jorgemoralespou]

I don't think end users understand what packages do internally, they only understand what values can be provided, and sadly, there's nowhere in any package that I have seen any warning saying "don't use an existing namespace" or it will be removed if the package is removed, and my experience with packages is quite extensive :-D