investigate whitelisting pods/exec for kdcluster pods

[joel-bluedata]

kdcluster pod containers are allowed to run as root because most of them are acting as "virtual Linux hosts" and need to e.g. run init as the entrypoint command to start up the usual Linux process tree.

If the environment running the kdclusters uses RBAC to block general users from doing "kubectl exec" then this isn't a concern. But if some users must be granted "kubectl exec" privileges it's a different story. E.g. what if such a user is malicious and knows a container-exit exploit. And even if you don't normally grant exec privileges to users, it would be good to have some confidence that if some process DOES end up granting exec privileges you don't have to worry about it meaning that the exec-ers will inadvertently be allowed to get into kdcluster pods.

There are various RBAC and/or policy (e.g. gatekeeper) ways to address this, with the goal of allowing "kubectl exec" only to the necessary pods and not to kdcluster pods. But these methods are all kind of tedious to manage in cases where you continually have new kdclusters and other pods being created, and potentially fragile to manage correctly. They also require a degree of "kubedirector awareness" to not break KD functionality; for example you can't just globally block exec into containers that are allowed to run-as-root, because KD itself needs to exec into such containers in order to run its script hooks.

It would be preferable for KD to help secure this situation.

KD could run a webhook to process any exec/attach attempts to kdcluster pods, and only allow them in certain circumstances... for example allow KD and system:masters to perform those actions, and probably also have some other admin-configurable whitelist. A gatekeeper policy could then (for example) be used to reject exec/attach elsewhere, or just blanket forbid non-kdcluster pods to run-as-root. This requires a little bit of KD awareness to exempt kdcluster pods from that policy (e.g. exempt pods with a "kubedirector.hpe.com/kdcluster" label) but nothing deeper than that.

[joel-bluedata]

I've got a proof of concept hacked up on the https://github.com/joel-bluedata/kubedirector/tree/exec-blacklist branch in my repo, just to verify that KD could do the exec filtering. I haven't yet thought deeply about how configurable the whitelisting should be.