Introduce automated flaky test tracking like OpenSearch

[YongGoose]

Why you need it?

I would like to suggest introducing an automated flaky test tracking mechanism, similar to what OpenSearch has adopted.

How it could be?

Proposal

Introduce a similar system to automatically report flaky tests in Seata. This would help:

• Systematically track flaky behavior
• Prioritize test stability improvements
• Avoid merging regressions masked by inconsistent test results

Implementation Ideas

• Hook into existing CI test result processing
• Identify tests that fail inconsistently across retries
• Auto-generate GitHub issues using a bot or GitHub Actions
• Apply consistent labels (e.g., flaky-test, test-failure, component:)

Benefits

• Easier triage of test failures
• More transparent test health history
• Helps contributors focus on test reliability

Thanks 🙏

Other related information

• https://learn.microsoft.com/en-us/azure/devops/report/powerbi/sample-test-analytics-flaky-tests?view=azure-devops&tabs=powerbi

[OmCheeLin]

Hey，could you assign it to me？

[YongGoose]

Hey，could you assign it to me？

Sure, Go for it 🚀

[YongGoose]

@OmCheeLin

This issue seems quite complex, so I think it might be a good idea to break it down into smaller sub-issues. What do you think?

If you agree, let's discuss together what sub-issues we could create.