[heartbeat] States and Improved Errors

[andrewvc]

Fixes https://github.com/elastic/beats/issues/32163

Adds a notion of state across checks, with flapping as a bonus.

Why is it important?

Checklist

• [ ] My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[mergify[bot]]

This pull request does not have a backport label. Could you fix it @andrewvc? 🙏 To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-09-13T20:37:39.513+0000

• Duration: 50 min 3 sec

Test stats :test_tube:

Test	Results
Failed	0
Passed	1857
Skipped	25
Total	1882

:green_heart: Flaky test report

Tests succeeded.

:robot: GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b tv2 upstream/tv2
git merge upstream/main
git push upstream tv2

[rdner]

@andrewvc is this PR still relevant?

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b tv2 upstream/tv2
git merge upstream/main
git push upstream tv2

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[andrewvc]

I'm realizing that this is missing one piece of functionality, which is geolocation awareness, which is pretty important. We should segment everything by geo. I'll add that soon, but I think it's still reviewable in its current state, I'll try and make the geo stuff into a single commit to make it easy to review.

[andrewvc]

Additional changes (added to top desc as well):

1. Disabled flapping for now, tests are in place but a hard coded value determines whether it is enabled now, it is currently set to false
2. New settings for geolocation added at heartbeat.location and per-monitor on location.
3. We now scope location lookups to geolocation (and type for efficiency)

Will add more tests and docs for location soon, hopefully tomorrow

[andrewvc]

This is now ready for a full review, having incorporated initial feedback from @dominiqueclarke .

It now has significantly enhanced functional-ish tests via many enhancements to the scenarios system / framework.

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b tv2 upstream/tv2
git merge upstream/main
git push upstream tv2

[andrewvc]

I believe I've addressed the incremental you reported so far @vigneshshanmugam thanks for starting on this review!

[andrewvc]

Tests are failing on win, blocked on https://github.com/elastic/beats/issues/32994 for now

[andrewvc]

@vigneshshanmugam FYI 6ecac3e fixes an interesting little bug, where heartbeat would try to connect to ES with the default options when non-ES outputs were defined. This was caught by the python tests

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b tv2 upstream/tv2
git merge upstream/main
git push upstream tv2