kyverno
[Feature] Support running flaky-tests N times before marking them failed
Problem Statement
We are planning to use chainsaw for our k8s testing. We have e2e tests written in go which do a lot of system testing using k8s. Since these tests work with a bunch of systems, things can sometimes go wrong due to flakiness in environment. For example, if API is being upgraded and the tests are trying to reach to API or requests are getting rate-limited etc. If a test fails due to some reason, we would like to re-run it automatically just to see if it passes the next time or not than requiring someone to manually trigger tests again. We would like to do this only for the tests which have failed. Chainsaw does have a flag --repeat-count=N but that always runs the test N number of times (irrespective of failure). We would like to have something similar to what ginkgo has (ginkgo.flake-attempts=N) so that flaky tests can be auto-retried.
Solution Description
Similar to --repeat-count, add a flag --flake-attempts or --repeat-failed-count which will make the failed test run max N times or until it succeeds.
Alternatives
No response
Additional Context
No response
Slack discussion
https://kubernetes.slack.com/archives/C067LUFL43U/p1733811046564419
Research
- [X] I have searched other issues in this repository and mine is not recorded.
