agentcfg: implement Elasticsearch fetcher

[axw]

Motivation/summary

Implement a new agentcfg.Fetcher which queries agent config from Elasticsearch directly, searching the .apm-agent-configuration index created by Kibana.

This enables running APM Server standalone, directly or with Elastic Agent, without requiring APM Server to communicate with Kibana to enable APM agent configuration.

Checklist

• [ ] Update CHANGELOG.asciidoc ~- [ ] Update package changelog.yml (only if changes to apmpackage have been made)~
• [ ] Documentation has been updated

How to test these changes

1. docker-compose up -d
2. apm-server -E output.elasticsearch.username=admin -E output.elasticsearch.password=changeme -E instrumentation.enabled=true
3. create agent configuration matching service name "apm-server"
4. curl http://localhost:8200/config/v1/agents?service.name=apm-server

The response should include the matching agent configuration.

Related issues

None

[mergify[bot]]

This pull request does not have a backport label. Could you fix it @axw? 🙏 To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-7.x is the label to automatically backport to the 7.x branch.
• backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[apmmachine]

:broken_heart: Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-07-21T11:44:04.735+0000

• Duration: 9 min 18 sec

Steps errors

Expand to view the steps failures

Run intake

• Took 5 min 48 sec . View more details here
• Description: ./.ci/scripts/intake.sh

:robot: GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /hey-apm : Run the hey-apm benchmark.

• /package : Generate and publish the docker images.

• /test windows : Build & tests on Windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify[bot]]

This pull request is now in conflicts. Could you fix it @axw? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agentcfg-elasticsearch upstream/agentcfg-elasticsearch
git merge upstream/main
git push upstream agentcfg-elasticsearch

[mergify[bot]]

This pull request is now in conflicts. Could you fix it @axw? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agentcfg-elasticsearch upstream/agentcfg-elasticsearch
git merge upstream/main
git push upstream agentcfg-elasticsearch

[mergify[bot]]

This pull request is now in conflicts. Could you fix it @axw? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agentcfg-elasticsearch upstream/agentcfg-elasticsearch
git merge upstream/main
git push upstream agentcfg-elasticsearch

[simitt]

Productionized in https://github.com/elastic/apm-server/pull/9720/