consider blocking service account deletion on app crs

[cppforlife]

currently if you deploy app cr and associated service account and delete them together, there is a race between kapp controller deleting workload and k8s deleting associated service account. once service account is deleted, kapp controller of course cannot delete app workload.

currently if you are deploying appcr+sa with kapp, you can add order rule; however, in a more basic cases (using kubectl for example), we should probably enforce "dependency" in api server via finalizer on service account and associated secrets. to support usage of same service account by multiple app crs, it's probably needed to add one finalizer per app cr to service account.

[danielhelfand]

One specific case to keep in mind with this issue is when a user deletes a namespace with an App and service account in it, the serviceaccount can get deleted before the App causing the whole deletion to not progress.

We'll raise this issue today in IPM and discuss if we can move forward with using a finalizer pattern to solve the problem.

[danielhelfand]

There are also other resources that would need to be factored into this solution:

• service tokens
• permission resources (Roles, RoleBindings)

It's unfortunately not enough to simply factor in the service account as part of this. So this issue unfortunately may need a different pattern than what is outlined above.

[benmoss]

To add to what Daniel was saying, Roles and RoleBindings are what give ServiceAccounts their permission to operate on resources, and so we'd have to have a way of preventing those from being deleted or modified as well to really have the desired functionality here. With the way that RBAC works it doesn't seem possible to know which Role/RoleBindings are relevant.

To think about this problem in another way, you give kapp-controller access to deploy this app using a user (service account). You then revoke that user's permissions (delete/modify rolebindings/roles), or delete the user entirely. It makes sense then that kapp-controller would not be able to delete or modify the app.

I don't know if it makes sense to try to disallow deleting the service account or roles/rolebindings. Kubernetes will let you run a pod as a serviceaccount and then delete the serviceaccount while the pod is still running.

I'm not sure what the relevance of service tokens are :).

[scothis]

Another approach is to grant kapp-controller permission directly to delete any resource in the cluster. And use those permissions when deleting an app instead of the user defined service account. This would only work in the same cluster.

[benmoss]

I think this was the approach that Helm used with Tiller, and was generally seen as a bad idea since it opened the door for privilege escalation