Remove docs and unit tests surrounding running agent as non-root

[naemono]

Since v8.16 Elastic Agent doesn't need the daemonset to adjust directory permissions when running as non-root. This removes/adjusts the documentation to make this more clear. It also removes the e2e test.

Testing

• Applied fleet-kubernetes-integration.yaml in the default namespace and everything came up without issues (see screenshot)
• Verified volume uses hostPath:

      volumes:
      - hostPath:
          path: /var/lib/elastic-agent/default/elastic-agent/state
          type: DirectoryOrCreate
        name: agent-data

This will be paired with a documentation PR as well, fyi.

[prodsecmachine]

:white_check_mark: Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
:white_check_mark:	Open Source Security	0	0	0	0	0 issues
:white_check_mark:	Licenses	0	0	0	0	0 issues

:computer: Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[pebrc]

Do you think there are security-conscious users who would prefer the init container pattern (short-lived elevated privileges) over a continuously running agent container with CAP_NET_RAW and CAP_CHOWN? Or am I overthinking this?

I think this is a valid point and I think the init container pattern has indeed advantages over running the main container with elevated privileges. You mentioned that CAP_SETPCAP and privilege escalation is required, that sounds like quite far reaching privileges to me.

[naemono]

Do you think there are security-conscious users who would prefer the init container pattern (short-lived elevated privileges) over a continuously running agent container with CAP_NET_RAW and CAP_CHOWN? Or am I overthinking this?

I think this is a valid point and I think the init container pattern has indeed advantages over running the main container with elevated privileges. You mentioned that CAP_SETPCAP and privilege escalation is required, that sounds like quite far reaching privileges to me.

I don't disagree with what is being said here. I struggled with how to present this such that is was "clear" to the user the approaches one could take with respect to Agent without making it a convoluted mess. Unfortunately with the following points

The agent container needs CAP_SETPCAP and CAP_CHOWN capabilities - these are enabled by default in all major container runtimes but a user can/may drop them. securityContext.allowPrivilegeEscalation should be true - IIRC this is also the default for most container runtimes but still a user can/may set this as false.

Things become even more difficult to document for Agent. I see some options:

1. Leave docs as-is. This unfortunately has the disadvantage of not even pointing out the 816 changes that many users may want to use, as it's a greatly simplified approach and handles the "defaults" for many/most users.
2. Present the 8.16+ option along-side the other options while trying to keep it clear the versions required (ECK >= 2.10.0 and Agent < 7.14.0), (ECK * and Agent 8.16+), etc for each approach.
3. Potentially other options...

Any preferences for an approach to try and bring this documentation up to current @pebrc @pkoutsovasilis ?

[barkbay]

The agent container needs CAP_SETPCAP and CAP_CHOWN capabilities [...] [...] I'm wondering if there's still value in the init container approach for some users - where only the init container runs with elevated capabilities to set up permissions, and then everything is dropped from the main agent container.

Sorry I'm a bit out of my depth here, and this is maybe off topic, but could you help me understand how running an init container could help avoid the need for CAP_SETPCAP?

[pebrc]

The agent container needs CAP_SETPCAP and CAP_CHOWN capabilities [...] [...] I'm wondering if there's still value in the init container approach for some users - where only the init container runs with elevated capabilities to set up permissions, and then everything is dropped from the main agent container.

Sorry I'm a bit out of my depth here, and this is maybe off topic, but could you help me understand how running an init container could help avoid the need for CAP_SETPCAP?

I think you might be right that Agent's main container needs CAP_SETPCAP regardless of the chowning happening in an init container because it needs to be able to setup its Beats child processes correctly. Maybe @pkoutsovasilis knows more here.

[pkoutsovasilis]

Sorry I'm a bit out of my depth here, and this is maybe off topic, but could you help me understand how running an init container could help avoid the need for CAP_SETPCAP?

The main point I'm trying to make is whether there's value from a security perspective for users who prefer an init container running as root to perform chowning, instead of a long-running elastic-agent container with securityContext.allowPrivilegeEscalation: true. My focus here is more on the privilege escalation setting rather than specific capabilities, but let me explain the capabilities mechanism as well:

• When transitioning from a rootful process (container runtime) to a non-root process (elastic-agent), the effective capabilities are not maintained. As documented in the Linux capabilities man page under "Effect of user ID changes on capabilities": when the effective user ID changes from 0 to nonzero, all capabilities are cleared from the effective set. However, a rootless process can elevate its capabilities if they are part of the bounding set (this is controlled by dropped and added capabilities) and privilege escalation is allowed (securityContext.allowPrivilegeEscalation: true). This is what elastic-agent does here when running as non-root - it elevates its effective capabilities to those in its bounding set.

  • Note that to raise capabilities in the Effective set, they must first be in the Permitted set, which elastic-agent achieves here by setting file capabilities.

• After elevating the capabilities in its effective set, and because elastic-agent spawns subprocesses, it needs to modify both the inheritable and ambient sets of its own. Modifying the ambient set requires CAP_SETPCAP, which happens here. This is necessary because inheritable capabilities are not preserved across execve when running as non-root, so such applications should use ambient capabilities to allow child processes to inherit them.

  • I think there's an assumption here that most probably holds true currently but may change: you assume that only the elastic-agent process with elevated privileges will perform all chowning, but this could easily change if an agent component, spawned as subprocess, needs to do the same. To complicate matters further, elastic-agent sometimes spawns subprocesses of its own binary that might need to perform chowning in the near future. In contrast, an init container approach could eliminate the need for CAP_SETPCAP because path ownership can be adjusted there without code changes. Again, these two capabilities CAP_SETPCAP and CAP_CHOWN are recommended for a rootless elastic-agent, not mandatory.

@barkbay does the above help?

I think you might be right that Agent's main container needs CAP_SETPCAP regardless of the chowning happening in an init container because it needs to be able to setup its Beats child processes correctly. Maybe @pkoutsovasilis knows more here.

@pebrc I don't think this is correct; actually, for versions prior to 8.16 there was no way for a rootless execution to elevate even its own capabilities. The same holds true for any rootless invocation that has securityContext.allowPrivilegeEscalation: false. As a matter of fact, elastic-agent has no issue running with securityContext.allowPrivilegeEscalation: false and all capabilities dropped if the state path is an EmptyDir and the user is running components that don't need any further capabilities. Also, AFAIK the component spec of elastic-agent has no field that captures which capabilities are needed by components, so the current approach is that components get whatever capabilities elastic-agent has.

Also, I may be missing something, but CAP_SETPCAP isn't particularly dangerous as a permission to have:

CAP_SETPCAP If file capabilities are supported (i.e., since Linux 2.6.24): add any capability from the calling thread's bounding set to its inheritable set; drop capabilities from the bounding set (via prctl(2) PR_CAPBSET_DROP); make changes to the securebits flags.

[pebrc]

@pkoutsovasilis what is you recommendation? We should keep the recipe around because there is value in having instructions how to run agent without privilege escalation?

[pkoutsovasilis]

@pkoutsovasilis what is you recommendation? We should keep the recipe around because there is value in having instructions how to run agent without privilege escalation?

yes this would be my proposal; let's extend our current documentation to capture that elastic-agent can be invoked rootless without an init container but this would require securityContext.allowPrivilegeEscalation: true and CAP_CHOWN capability if the user needs to have the state persisted (we can debate if CAP_SETPCAP is required, I would leave it in to guarantee that every subprocess gets the capabilities). Then shift a bit the existing documentation to point out that there is another way to run rootless elastic-agent without privilege escalation.

[naemono]

Thanks everyone for the discussions here on the best path of the documentation. I'll work towards extending the documentation to take into account all paths to run Agent and ask for reviews when this is complete.