allure-framework
JUnit repeated tests are shown as separate tests in the report (not as retries)
Describe the bug We have a buch of unstabe tests that we execute with a re-run. For that, we use @RepeatedTest (org.junit.jupiter.api) or @RetryingTest (org.junitpioneer.jupiter) (the latter re-runs a test for n times only if the test is failing).
But whenever a re-run of the same test is done, the report shows it not as a retry, but as n separate tests.
Screenshots
Environment (please complete the following information):
| Allure version | 2.21.0 |
|---|---|
| Test framework | [email protected] |
| Allure adaptor | [email protected], [email protected] |
| Generate report using | [email protected] |
It is implemented this way because Allure always shows the last result, hiding the previous as retries. Still, for repeated tests, users usually need to highlight any failures in any repetition. So disabling the retries for repeated tests is the best we can do. Otherwise, there is no point in using such tests at all
The thing is, for example, org.gradle.test-retry plugin works as expected with the reports: the retried test is overall marked as passed, although marking it as "changed after retry" and showing the retries grouped.
In the meantime, the issue I mention above is that the test retries are not grouped - Allure considers them as separate tests.
Please note, the problem described above is also reproducing via the Wedriver.IO Retry Flaky Tests functionality. Allure considers retried tests as separate tests.
@RepeatedTest implemented this way intentionally. Otherwise, Allure will only show the latest result, removing any point to use repeated tests.
the problem described above is also reproducing via the Wedriver.IO Retry Flaky Tests functionality. Allure considers retried tests as separate tests.
Please create a separate discussion for that (https://github.com/orgs/allure-framework/discussions)
@baev hi! I understand why @RepeatedTest should be considered as separate tests (this annotation makes explicit retries with no regard to the result).
But what about @RetryingTest (org.junitpioneer.jupiter) ?
I think this is a good annotation to use with flaky tests, and can be considered when grouping tests in the report as retries of the same test.
https://junit-pioneer.org/docs/retrying-test/