Security Concerns - Node Capabilities and Root

[Jmoore1127]

The cluster pods which are created as part of the VerticaDB CR require node kernel permissions as seen here: https://github.com/vertica/vertica-kubernetes/blob/1733182b50a7e86ccf013a4277c633c79673c4f4/pkg/builder/builder.go#L459-L466

The comments say that it is for ssh access to the pods in OpenShift. Is there a way to remove those for k8s? Having them forces us onto isolated nodes since root combined with kernel perms presents a much larger risk surface within our cluster. If we could remove those we could run on dedicated nodes only which is a much simpler deployment.

Similarly, the pods need root access so the operator run sudo inside them. It seems odd that the operator is taking imperative actions on the pods rather than leaning into the state seeking, reconciliation behaviors of kubernetes. Ideally, we'd like to see the operator taking actions on the k8s api instead (mounting configmaps, coordinating pod restarts, etc) which would eliminate the need for external root access.

I imagine those present large changes to how things currently work. However, with this large of a risk surface it has raised a lot of questions as to whether this operator is actually production ready or not.

I would be willing to help out with some of these changes but I am not a Vertica expert.

[spilchen]

We are going to be working on removing the ssh requirement within Vertica. ssh is currently used as the communication mechanism between nodes for admintools. We are working on replacing that with a management SDK that utilizes REST endpoints in the server. This is a large effort and won't be fully in place until sometime next year.

With ssh removed, we will be able to remove the SYS_CHROOT and AUDIT_WRITE capabilities and the need to run sudo.

As for the block of code you highlighted. We added it for OpenShift because it was needed in order to stand up ssh with their security context contraints (SCC). The same capabilities are needed for non-OpenShift, but we didn't need to explicitly define them. I'm guessing the default security policy allowed it. I need to dig into this part a bit more.

[Jmoore1127]

Wanted to check back in after a few months and see if this was still on the roadmap after the new year. Still very interested in getting this fixed.

[spilchen]

Yes, this is still very much on our roadmap. A team should start working on the admintools replacement in the next month or so. You should see quite a few different contributors to this repo as we ramp up.

[spilchen]

I'm providing an update on this issue. A few weeks ago, we released a new server (24.1.0) and operator (2.0.1) that significantly reduced the reliance on root capabilities.

We've introduced a new deployment method called vclusterops, which replaces admintools and communicates via REST calls instead of SSH. This method is the default for the new v1 API of the VerticaDB CR. When deployed in this manner, all sudos have been removed. However, you may still find some sudo calls in the operator code when deployed with admintools.

There is one remaining issue that we're working on in the current server release. The server container uses s6-overlay as an init process, which requires setuid privileges. We're transitioning to a model where we'll run an admin process in a sidecar called the node management agent (NMA). This change will eliminate the need for the init process and setuid privileges.

I'll keep this issue open until the server change is released in April with the 24.2.0 server release.

[spilchen]

As version 24.2.0 was released last Friday, I am closing this now as all root security concerns have been addressed. https://docs.vertica.com/24.2.x/en/new-features/24.2/containers-kubernetes/#support-openshift-restricted-v2-scc