Add heartbeat support for V2

[fearful-symmetry]

What does this PR do?

This PR adds V2 support for heartbeat running under elastic-agent.

I haven't thoroughly tested this yet, but I wanted to put in the PR while I juggle the 2-3 other things I'm working on. See the Auditbeat PR for more detailed instructions on how to test: https://github.com/elastic/beats/pull/33026

Here's an example of an elastic-agent input config that should work with heartbeat:

- id: synthetics/http-synthetics-f42c9747-2455-491b-a66c-967adc98007a
    name: test
    revision: 1
    type: synthetics/http
    use_output: default
    meta:
      package:
        name: synthetics
        version: 0.10.2
    data_stream:
      namespace: default
    package_policy_id: f42c9747-2455-491b-a66c-967adc98007a
    streams:
      - id: synthetics/http-http-f42c9747-2455-491b-a66c-967adc98007a
        name: test
        type: http
        enabled: true
        data_stream:
          dataset: http
          type: synthetics
        urls: 'http://localhost:9200'
        schedule: '@every 3m'
        timeout: 16s
        max_redirects: 0
        response.include_headers: true
        response.include_body: on_error
        check.request.method: GET
        processors:
          - add_observer_metadata:
              geo:
                name: Fleet managed
          - add_fields:
              target: ''
              fields:
                monitor.fleet_managed: true

Checklist

• [x] Test under agent
• [x] My code follows the style guidelines of this project
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-10-10T22:13:08.191+0000

• Duration: 39 min 41 sec

Test stats :test_tube:

Test	Results
Failed	0
Passed	313
Skipped	0
Total	313

:green_heart: Flaky test report

Tests succeeded.

:robot: GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[fearful-symmetry]

Alright, finally managed to test this.

[paulb-elastic]

Pinging @emilioalvap and @andrewvc as there was some discussion about this regarding https://github.com/elastic/beats/issues/32163

[andrewvc]

@fearful-symmetry with the V2 protocol does the output.elasticsearch section still get written? https://github.com/elastic/beats/issues/32163 , which was merged recently, depends on it, reading back from ES to get the last known state for heartbeat

[andrewvc]

You can see where we set this up, and how, here: https://github.com/elastic/beats/blob/main/heartbeat/beater/heartbeat.go#L72

[fearful-symmetry]

@andrewvc

with the V2 protocol does the output.elasticsearch section still get written?

Yes, it does. Until the shipper comes along, we're still using built-in elasticsearch output. Theoretically, nothing should change in terms of the actual configs that are sent to individual components of a beat; this V2 support mechanism is just a compatibility shim for V2.

[fearful-symmetry]

You can see where we set this up, and how, here: https://github.com/elastic/beats/blob/main/heartbeat/beater/heartbeat.go#L72

@andrewvc does this work under the current fleet setup? This happens outside of the reloaded registers here: https://github.com/elastic/beats/blob/main/heartbeat/beater/heartbeat.go#L213, which means that the code you linked to (I think) won't respond to changes in the fleet config, regardless of this PR.

[andrewvc]

@fearful-symmetry I'm realizing it's only been tested on the beat, thanks a bunch for noticing that, I've opened https://github.com/elastic/beats/issues/33244 to track.