Add logic for beats `User-Agent`

[fearful-symmetry]

Proposed commit message

This adds a bit of logic to change the beats ES connection user agent if the beat is running under agent.

The original issue doesn't suggest what we should change the User Agent string to, but this was simple enough that I figured it would be faster to put in a fix and we can decide in the PR; currently, it just appends -Managed to the string.

Checklist

• [x] My code follows the style guidelines of this project
• [x] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [x] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related

Closes https://github.com/elastic/elastic-agent/issues/3065

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[mergify[bot]]

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @fearful-symmetry? 🙏. For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2024-05-03T22:00:05.623+0000

• Duration: 112 min 42 sec

Test stats :test_tube:

Test	Results
Failed	0
Passed	29427
Skipped	2061
Total	31488

:green_heart: Flaky test report

Tests succeeded.

:robot: GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[amitkanfer]

Did we also want to add --unprivileged to the User Agent string (if running in that mode)? @pierrehilbert

[strawgate]

Not to pile on... But distinguishing managed from standalone might be nice too...

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[pierrehilbert]

@amitkanfer As we are already having this information in the local metadata, do you have a specific case you want to cover by adding it to User-Agent? For Telemetry only?

@strawgate We discussed this quickly during our meeting on Thursday and I think this is a great idea to user the User-Agent for that! @fearful-symmetry WDYT?

[fearful-symmetry]

@pierrehilbert I think that's a good idea. I'm not sure if there's any way for beats to know what mode agent is running under? Never heard of anything like that. If we're interested in telemetry, I wonder if we can add that string to some headers sent by agent itself somehow?

[jlind23]

@fearful-symmetry @pierrehilbert can we piggy back on this to also tackle https://github.com/elastic/elastic-agent/issues/3065?

[fearful-symmetry]

@jlind23 that issue seems largely the same as https://github.com/elastic/ingest-dev/issues/3202, with the addition of also reporting the agent version. I'm not sure it's possible for the agent version to differ from the beats version?

[cmacknz]

I'm not sure it's possible for the agent version to differ from the beats version?

For now and with independent releases they'll always be the same major.minor, the patch may differ.

[cmacknz]

Not to pile on... But distinguishing managed from standalone might be nice too...

The Beats today have no way to know this as they do not interact with Fleet. There are a couple of fields we could use to try to infer it but that would be fragile. We would have to add this information to the control protocol or start injecting it on the Beat command line. This should be tackled as a separate issue and PR if we decide to do it.

Beats know they are running under Elastic Agent, but not much else.

[fearful-symmetry]

So, the AgentInfo struct in the protobuf spec sends the agent version, which means that shouldn't be too hard to add to the metadata. Agent state (fleet or self-managed) might be harder though. We might have to expand the protobuf spec to add that?

[strawgate]

We would have to add this information to the control protocol or start injecting it on the Beat command line. This should be tackled as a separate issue and PR if we decide to do it.

@cmacknz Given the way we intend to use this I do believe we should implement the planned user agents in a single release instead of spanning several releases

[cmacknz]

Agent state (fleet or self-managed) might be harder though. We might have to expand the protobuf spec to add that?

Yes it could go there or we could change the management.enabled command line parameter to take the values standalone or fleet instead of just true. I think we need a separate issue to implement this.

We could also pass the privileged state down if we really wanted to, I would not re-implement detection of that in agent.

I am going to create a separate issue for implementing these additional fields, and per https://github.com/elastic/beats/pull/39403#issuecomment-2096275443 above we shouldn't merge this change until they are implemented.

[cmacknz]

Created https://github.com/elastic/elastic-agent/issues/4683 to track including the fleet/standalone state and privilege level. Including whether an agent is unprivileged is cheap to do if we are already in the code adding whether we are fleet managed, so let's just include it.

[fearful-symmetry]

Alright, so it seems like the rough order of operations is:

A) update the protobuf spec to add the needed metadata B) update the agent to actually send the metadata C) update the code beats-side

[cmacknz]

Correct

[strawgate]

Are we intending on passing agentbeat as the user agent for agent beats? Or are we setting the user agent based on which subcommand was chosen?

If it's agentbeat we likely lose quite a bit of fidelity?

[fearful-symmetry]

@strawgate based on what I've seen, under agentbeat the user-agent will continue to report the underlying beat that agent is controlling.

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b beats-user-agent-managed upstream/beats-user-agent-managed
git merge upstream/main
git push upstream beats-user-agent-managed

[fearful-symmetry]

Alright, PR updated with all the feature requests; elastic-agent PR incoming...

[fearful-symmetry]

I think the CI errors are unrelated?

[pierrehilbert]

Looks like we are all green. @belimawr could you please do another review?