[ci][flaky] reporting

[v1v]

What

Add flaky test reporting on a PR basis.

Tasks

• [ ] Enable ES indice.

[trentm]

Is there a link to some description of how flaky tests are identified? I see https://github.com/elastic/apm-pipeline-library/pull/599 added that functionality, but I don't see anything there that describes design or intent.

[apmmachine]

:broken_heart: Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-01-20T10:07:20.603+0000

• Duration: 22 min 43 sec

• Commit: 596af7997476b5e89250eb760c60ffb5a61b3f5d

Steps errors

Expand to view the steps failures

Show only the first 10 steps failures

Run Tests

• Took 2 min 3 sec . View more details here
• Description: .ci/scripts/test.sh -b "release" -t "" "10"

Archive the artifacts

• Took 0 min 0 sec . View more details here
• Description: [2022-01-20T10:25:50.642Z] Archiving artifacts [2022-01-20T10:25:50.930Z] ‘test_output/*.tap’ d

Run Tests

• Took 4 min 17 sec . View more details here
• Description: .ci/scripts/test.sh -b "release" -t "" "10.0"

Run Tests

• Took 2 min 57 sec . View more details here
• Description: .ci/scripts/test.sh -b "release" -t "" "10.0"

Archive the artifacts

• Took 0 min 0 sec . View more details here
• Description: [2022-01-20T10:25:47.060Z] Archiving artifacts [2022-01-20T10:25:47.381Z] ‘test_output/*.tap’ d

Run Tests

• Took 4 min 12 sec . View more details here
• Description: .ci/scripts/test.sh -b "release" -t "" "10.0"

Run Tests

• Took 2 min 59 sec . View more details here
• Description: .ci/scripts/test.sh -b "release" -t "" "10.0"

Archive the artifacts

• Took 0 min 0 sec . View more details here
• Description: [2022-01-20T10:25:12.357Z] Archiving artifacts [2022-01-20T10:25:12.660Z] ‘test_output/*.tap’ d

Windows Batch Script

• Took 1 min 1 sec . View more details here
• Description: node test/test.js

Error signal

• Took 0 min 0 sec . View more details here
• Description: hudson.AbortException: script returned exit code 1

:grey_exclamation: Flaky test report

No test was executed to be analysed.

:robot: GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run module tests for <modules> : Run TAV tests for one or more modules, where <modules> can be either a comma separated list of modules (e.g. memcached,redis) or the string literal ALL to test all modules

• run benchmark tests : Run the benchmark test only.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[v1v]

https://github.com/elastic/apm-pipeline-library/pull/754 provides a higher overview of the aim of this particular flaky reporting in GitHub on a PR basis.

Regarding how flaky tests are identified, that's documented in an internal repo, I'll send you those details in a bit.

[trentm]

@v1v Given we are moving away from Jenkins, can we close this PR?

[trentm]

Closing this. @v1v please feel free to re-open if there is something to salvage here.