[DX Streamline] Establish flaky test eradication program

[BigLep]

Done Criteria

1. There is a metric that shows how much lotus is plagued by test flakiness. Eradication of flaky tests should have an impact on this metric.
2. It is self-service for anyone to find this metric.
3. It is self-service for anyone to find flaky tests and how flaky they are.
4. There is a public SOP/runbook on how a maintainer should handle when a flaky test is discovered.

Why Important

Flaky tests are insidious. We have them in our codebase, and it has two primary problems:

1. Slows down the velocity we can merge because judgement needs to be applied for whether it's safe to merge and we need to rely on those with admin permissions (e.g., maintainers) to merge.
2. It's compounding. Once we have some, it's easier for more to slip in, which makes it harder to dig out later.

We want to have the standard of "we only merge if CI is green", and that is much easier bar to hold if flakiness has been eradicated.

User/Customer

Maintainers and contributors.

Notes

• This issue can be completed even before all flaky tests are eradicated. This issue is done when we have credible visibility and plan to get there.
• It's acknowledged that handling flaky tests is not a new effort for Lotus development. This issue is more about building on existing efforts and getting more robust so that we win this battle.
• Lotus has the "impact/test-flakiness" label for historically identified flaky tests.
• Handling flaky tests is nothing unique to Lotus. There is lots of prior art and tooling. Some examples / discussion I'm aware of:
  • go-libp2p: they have a score badge on their readme which links to deeper analysis at https://marcopolo.github.io/FlakyTests/
  • Observable notebooks have been created concerning this topic: https://observablehq.com/@observablehq/integration-test-flakiness
  • Github projects: https://github.com/F-Secure/flaky-tests-detection
  • Articles on the topic:
    • https://engineering.fb.com/2020/12/10/developer-tools/probabilistic-flakiness/
    • https://dl.acm.org/doi/10.1145/3377813.3381370
  • Dedicated service/tool: https://buildpulse.io/