Agentbeat tests based on agentbeat.spec.yml

[oakrizan]

Proposed commit message

Added checks for Agentbeat that execute all unique commands for inputs from agentbeat.spec.yml and validates that those can start without failures.

Checklist

• [ ] My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Related issues

• Relates: https://github.com/elastic/ingest-dev/issues/3746

Logs

Buildkite builds: https://buildkite.com/elastic/beats-xpack-agentbeat/builds?branch=oakrizan%3Aagentbeat-integration-tests-spec

[mergify[bot]]

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @oakrizan? 🙏. For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify[bot]]

backport-8.x has been added to help with the transition to the new branch 8.x. If you don't need it please use backport-skip label and remove the backport-8.x label.

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agentbeat-integration-tests-spec upstream/agentbeat-integration-tests-spec
git merge upstream/main
git push upstream agentbeat-integration-tests-spec

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agentbeat-integration-tests-spec upstream/agentbeat-integration-tests-spec
git merge upstream/main
git push upstream agentbeat-integration-tests-spec

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

This is the first part of a solution, the spec parsing works, but we aren't actually testing enough.

When I run this locally, I see it starting agentbeat with each of the beats as sub-commands. However, each of these Beats is configured to do nothing, so we aren't testing as much as we should.

The example of the type of problem we want to catch is https://github.com/elastic/beats/pull/39818, where agentbeat was silently not including an import that was included if you ran it directly as part of heartbeat.

The minimal reproduction for this is this heartbeat configuration which I'll put in heartbeat.yml:

heartbeat.monitors:
- type: browser
  enabled: true
  id: my-monitor
  schedule: '@every 10s'

output.console:
  hosts: ["localhost:9200"]
  enabled: true

Then if I revert 4c4b2f899587cdb251b2c1d947395eb7664842bd and run the following it reproduces:

git revert 4c4b2f899587cdb251b2c1d947395eb7664842bd
cd x-pack/agentbeat
mage build
./agentbeat heartbeat -c ./heartbeat.yml # Contains the heartbeat config above
Exiting: could not create monitor: factory could not create monitor: monitor type browser does not exist, valid types are [icmp synthetics/icmp tcp synthetics/tcp http synthetics/http]

What we need to do is to parse the list of inputs out of the spec files, and then create a config file for each of the Beats that includes all the inputs it can run, and then make sure that the Beat starts without exiting like this.

The config file for each Beat looks slightly different, see https://github.com/elastic/beats/blob/main/x-pack/filebeat/filebeat.yml for example compared to the heartbeat example above. Probably templating the config file would work for this.

Additionally, it would be nice if these tests were written as Go tests. Then the test report would be in the same format as all the other tests. At least a test case per beat, you could have sub-tests for each of the inputs.

[pazone]

What's the status of this PR?

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agentbeat-integration-tests-spec upstream/agentbeat-integration-tests-spec
git merge upstream/main
git push upstream agentbeat-integration-tests-spec