junit5
junit5 copied to clipboard
junit-team
Allow @RepeatedTest to stop upon failure
Many a times when a particular test is a failing once in a while (i.e. not consistently), I use the @org.junit.jupiter.api.RepeatedTest with a relatively high number like 10:
@RepeatedTest(10)
and trigger the tests to try and reproduce the issue. Some of these 10 runs will fail (as expected). However, the only reason I used the @RepeatedTest was to reproduce a failure and once that failure has occurred, I'm not really interested in running the remaining number of "count" that was passed to it.
Would it be a good idea to introduce something like stopOnFailure attribute (which can be set to true, but defaults to false) to this @RepeatedTest so that it considers the value count as the maximum number of times to run the test and if stopOnFailure is set to true then stops as soon as the first run within those 10 runs fails?
Tentatively slated for 5.6 M2 solely for the purpose of team discussion
@antimpatel Please note that we haven't taken a decision whether we want to add and maintain this functionality. However, it would certainly help to see it spelled out in a spike. I think one of the main complications will be to change the execution mode to same_thread if the feature is used. You'll probably have to add a getExecutionMode() method or sth. similar to TestTemplateInvocationContextProvider.
I like this idea. The option to stop on failure could help test runs terminate quicker if there are faults.
really useful, we also have such huge integration tests failing sporadically and needing 20 repetitions to prove that there is no problem, but after the first failure there is no need to run the remaining repetitions. My suggestion is: adding the number of failures to RepetitionInfo and let the test decide if it needs more repetitions.
JUnit Pioneer has the opposite: @RetryingTest stops upon the first success.
@qeepcologne Would you like to bring up the issue there?
Currently repeated test can be represented as individual tests. However failing early changes the semantics significantly. This introduces the concept of an "attempt" which is effectively a sub-test entity.
This has provided some unresolved conceptual difficulty in Cucumber and is likely to do so for JUnit 5. I've written more about it here: https://github.com/junit-team/junit5/issues/2925#issuecomment-1137837182.
This would be really useful to have!
We have CI runners of many different shapes and sizes, and for types of tests we need to repeat a test a certain amount of times to trigger a failure. For some runners, fewer repetitions are needed and for others, a lot more repetitions are needed. Stopping on the first failure means we could set a high repetition count without penalizing all CI runs (and runs in local development environments).
