ECK resources Helm chart - Elastic Agent & Elastic Fleet Server Agent

[naemono]

related to https://github.com/elastic/cloud-on-k8s/issues/5505

This PR contains the 2nd version of the ECK-managed resources helm charts including

eck-elastic-agent chart for installing elastic-agent eck-fleet-server for managing fleet server as an Elastic Agent minor updates to the eck-stack chart.

~~Note that this Chart allows multiple instances of Elastic Agent to be installed, along with allowing multiple instances of serviceAccounts, clusterRoles, and clusterRoleBindings.~~

cc @jmlrt @Kushmaro @framsouza

~~In Progress~~

• [x] additional manual testing
• [x] additional chart unit tests

[naemono]

make the whole RBAC side of deploying agent easier with the Helm chart

I would love to make this easier for the user, but I'm not sure the Helm chart is the right place to do it. If we wanted to ensure that rbac rules were sufficient to the configuration, that logic would likely make more sense, and be easier to maintain in the operator itself, as complex logic in Helm charts just comes across as complex, ugly, and difficult to maintain.

I can certainly go down that road and attempt this within the Helm charts. Please let me know and I'll make the changes, and see what they look like.

[pebrc]

Please let me know and I'll make the changes, and see what they look like.

I don't have something that is ready to implement. I feel like Agent is special enough, compared to Kibana and Elasticsearch, to warrant doing a bit of design work for the Helm chart.

For example one idea I had was to make two different Helm charts one for Fleet Server and one for Elastic Agents. Behind the scenes it is the same CRD of course still.

That would allow us to configure two different sets of default RBAC permissions.

More far reaching permissions for Agent, roughly oriented maybe on the Kubernetes observability recipe or maybe the DaemonSet published by the Agents team

And less permissions for the Fleet server chart.

User could then override the default RBAC permissions for each of these charts to either restrict them further (e.g. for an APM integration we don't need all the K8s API permissions that K8s observability needs) or expand them where necessary.

With a Fleet server chart and an Elastic Agent chart we should be able to cover most if not all use cases we have currently in the recipes section of the repository. This is just an idea at this stage and not ready to implement. Probably worth discussing with the team. My motivation is to find a solution how we can keep the templating logic minimal in the Charts but still add value for our users by starting with sane defaults that cover the most common case.

[naemono]

We met and had a discussion about the current implementation, vs a more specialized implementation for, say, Fleet, and comparing what the user experience was like between the two, we decided to implement the more specialized Helm chart for Fleet, which simplifies what the Agent helm chart looks like. I'll move this back to a WIP, and update when there's more information.

[naemono]

@pebrc Update: Both Elastic Agent, and Elastic Fleet Server Agent helm charts are now included in this PR, and are ready for review.

[thbkrkr]

Is it intentional that labels/annotations are managed differently in the ClusterRoleBinding/ClusterRole/ServiceAccount between the two charts? The eck-agent chart supports dedicated labels/annotations but for labels we also get the labels of the agent, while for the eck-fleet-server no annotation at all and only the labels from the fleet-server.

[naemono]

Is it intentional that labels/annotations are managed differently in the ClusterRoleBinding/ClusterRole/ServiceAccount between the two charts? The eck-agent chart supports dedicated labels/annotations but for labels we also get the labels of the agent, while for the eck-fleet-server no annotation at all and only the labels from the fleet-server.

This absolutely was not intentional, and has been resolved. Thanks for catching that @thbkrkr. I think I'll also add additional tests around labels/annotations similar to https://github.com/elastic/cloud-on-k8s/pull/6004/files#diff-9612072d3cca4f0281e66e5bd8fdb713f35792862686355d89ff507348419028R41

[naemono]

@thbkrkr While adding tests, I noticed that .Values.[labels|annotations], and .Values.[serviceaccount|clusterRole|clusterRoleBinding].[labels|annotations] weren't both making it into the rendered templates, and this was fixed, and tests were added for all.