dd-trace-java
dd-trace-java copied to clipboard
[POC] Flaky test retries
What Does This Do
- Use Develocity Test Retry (docs) to retry flaky tests in CI.
- Only tests marked as
@Flakywill be retried. - Failed runs are visible in the logs, and preserved. This keeps observability of intermediate runs in Datadog Test Optimization.
Why?
This would allow to get closer to CI pipelines always succeeding on the first run. Our past strategy seggregating flaky test to a special CI job that never fails the pipeline has led to many tests regressing from flaky to broken.
While retrying can make some individual jobs slower, if we account for the overall time between pushing and manually retrying, the overall time spent in CI should be way lower.
Compared to the old @Retry approach, this plugin has the advantage of not hiding intermediate failed runs. So we should still be able to monitor flaky tests. in fact, given that flaky tests are always retried at least once, flaky test detection should be more accurate, both at Circle CI and Datadog Test Optimization.
Caveats
There are some limitations of this plugin:
@Flakyannotation only works at class level, not method level (feature request).- It does not support advanced runtime conditions, such as filtering by JVM vendor or version, or limiting retyring to specific errors (feature request).
Contributor Checklist
- Format the title according the contribution guidelines
- Assign the
type:and (comp:orinst:) labels in addition to any usefull labels - Don't use
close,fixor any linking keywords when referencing an issue.
Usesolvesinstead, and assign the PR milestone to the issue - Update the public documentation in case of new configuration flag or behavior
Jira ticket: [PROJ-IDENT]