elastic
integrationservertest: Add detailed reporting to Slack
Current Slack message that is reported whenever the tests pass / fail is very basic.
Implement a detailed breakdown (perhaps in the form of a table) that shows the tests that were run as well as their status, so that we can easily tell which tests failed, which tests were run (were they expected?), whether we accidentally skipped some tests etc.
Example:
| Test | Status |
|---|---|
| Upgrade 8.15 -> 8.16 -> 8.17 | ✅ |
| Standalone-to-managed 7.17 -> 8.19 -> 9.1 | ❌ |
| Some other test | SKIP |
Did some preliminary research on this, and it seems to be more effort as initially thought.
Slack messages don't support Markdown
We would need to find another way to represent the tests and status etc. One way is to use Slack blocks API to send a list instead. See example via Slack block kit builder.
GitHub workflow matrix job outputs complication
According to this GH discussion: https://github.com/orgs/community/discussions/26639, there is no support currently for accessing a matrix job variables etc. from another job. The community has instead suggested using artifacts to pass matrix job outputs. So we will need to:
- Run matrix jobs
- For each matrix job, write outputs to a UNIQUELY NAMED file and upload as artifact
- After jobs are done, download all output files and construct Slack message for notification