gotestyourself
`--rerun-fails` and `--junitfile`: tests marked as failed even though they were rerun with success
Example: there are 178 tests in the test suite. We run them with the --rerun option, and when a test fails then the reporter adds it as failed in the jUnit report. But if succeeded during a re-run, then it's not marked as passed again.
Action URL: https://github.com/testcontainers/testcontainers-go/actions/runs/3198462957
Screenshot in case the build is discarded:
Thanks for raising this issue!
Indeed both the summary printed by gotestsum and the junit.xml report will include the failed tests that were re-run. They are included because they indicate a flaky test, and the team responsible for the tests most likely wants to know that despite the tests passing eventually, some did need to be re-run, so that they can fix the flaky test. The goal of --rerun-fails is not to hide flakes, but to make them less of a nuisance when a developer is trying to get other work done.
Is the error in the report causing a problem? I had a look at the build and it matches my expectation. The build is a success, but there were failed tests. Maybe gotestsum could add a line to the junit.xml to indicate the test will be re-run. Or maybe test-summary/action could indicate this somehow when it sees the overall status is success?
seems valuable to separate out or otherwise label those from non-flakey failures, in a scenario where a CI job fails, being able to immediately know which test(s) are causing the consistent failures and likely need to be addressed first vs intermittent failures that might be a lower priority is useful. It was definitely surprising to me to see a passing CI job with failed tests listed, especially if you are watching the logs of the job and can't see overall pass/fail of the job yet.
wrt junit xml, i'm not sure how consumers of the format handle or expect retries but i imagine its inconsistent between different tools, maybe a gotestsum option to only report consistent failures would be an easy way to avoid downstream problems if they occur..
Our team's use case for --rerun-fails is to get our CI pipeline to pass, despite the flakey tests. If the junit report contains failures, then the CI system will interpret those as failures, marking the CI job as failed instead of passed.
I certainly see the value of exposing which tests are flakey, but in this particular case I would like that to be in a different report rather than the junit report.
If the junit report contains failures, then the CI system will interpret those as failures, marking the CI job as failed instead of passed.
Interesting! Which CI system is this? All the CI systems I've worked with use the exit code of the process to determine pass or fail. That is generally a much more reliable way to determine success.
The specific system is GitLab, but most (all?) CI systems have a way to parse a JUnit XML report and then pass/fail the job based on those results.