junit5
junit5 copied to clipboard
junit-team
Introduce support for retrying failed flaky tests
Overview
Feature request to add native support to rerun failing flaky tests
Feature request
Is there any plan to support the ability of rerunning failed flaky tests? For example, it would be great to have something like @Flaky(rerun=3), which would rerun a failing flaky test up to n times.
In the past, in Surefire/Failsafe we could use rerunFailingTestsCount. However, it does not work in JUnit 5. As JUnit 5.0.0 was released nearly 1 year ago, it is unclear if and when it will be supported in Maven.
And, even if it will be supported in Maven in the future, it could be good to have a special annotation to specify that only some tests are expected to be flaky, and not all of them.
At the moment, the workaround is to stick with JUnit 4, which is a kind of a shame as JUnit 5 has a lot of interesting features :( (which I can only use in projects with no flaky tests)
Related SO question.
No, to my knowledge there are not currently any plans to implement such a feature.
But... I think it's a very good idea.
I've often wanted something like Spring Retry's @Retryable support for test methods. Of course, Spring Retry supports a much larger feature set than we would ever need/want in JUnit Jupiter.
Thanks for providing the link to the discussion on Stack Overflow. There are two interesting implementations there that can serve as inspiration.
Ideally, I'd like to see something like a @RetryableTest annotation that allows one to specify:
- maximum number of retries
- configurable exceptions that initiate a retry, defaulting to
{Throwable.class} - potentially a minimum success rate
@junit-team/junit-lambda, thoughts?
configurable exceptions that initiate a retry, defaulting to
{Throwable.class}
Hmm, how about defaulting to Exception.class or Throwable.class-except-blacklisted-exceptions?
My thinking is that in the off-chance an OutOfMemoryError is thrown, it should really be propagated to allow the JVM to shutdown as cleanly as it possibly can. :thinking:
I don't like flaky tests (read: checks). Who does? I don't like the idea of "fixing" flaky tests by (naiv, smart, conditional, [what|for]-ever) re-execution. I don't want to support that in Jupiter.
Easy work-around:
@Test void test() { REPEAT flaky() UNTIL flaky-signals-green OR break-out }
boolean flaky() { ... return true : false ... }
Perhaps (!) such an annotation could live in an extension providing library: like https://junit-pioneer.org or another one. If we need to expose a new extension point tailored to make the "retry" functionality possible, well, I'm fine with that.
@sormuras I do not like flaky tests either :) but unfortunately they are a reality :(
not an issue with unit tests (unless you work with randomized algorithms), but they are a big problem for E2E tests. There, it is hard to escape from flakyness. Even if a test wrongly fails 1 out 1000 times, when you have hundreds/thousands of E2E tests, there is a high probability that at least 1 fails, failing the whole build.
Of course one can use a try/catch in a loop, but that is a lot of boilerplate that has to be repeated each time for each test.
In the past, it was not an issue with JUnit 4, as this was externally handled with Maven. But this is not the case any more. In my case, this is a major showstopper for using JUnit 5. In some projects we tried JUnit 5 and were forced to revert to 4. In others, we use JUnit 5 for unit tests, but then have a (Maven) module for the E2E using JUnit 4.
Coincidentally, I recently hacked an extension that repeats failed tests:
class RepeatFailedTestTests {
private static int FAILS_ONLY_ON_FIRST_INVOCATION;
@RepeatFailedTest(3)
void failsNever_executedOnce_passes() { }
@RepeatFailedTest(3)
void failsOnlyOnFirstInvocation_executedTwice_passes() {
FAILS_ONLY_ON_FIRST_INVOCATION++;
if (FAILS_ONLY_ON_FIRST_INVOCATION == 1) {
throw new IllegalArgumentException();
}
}
@RepeatFailedTest(3)
void failsAlways_executedThreeTimes_fails() {
throw new IllegalArgumentException();
}
}
If JUnit 5 decides that flaky tests have no place in a proper testing framework, we pioneers don't have such scruples. :wink:
Nope. :laughing: Most extensions I write come from excursions into the extension model, not from actual use cases (hence the lack of research).
Looks like @arcuri82 has at least one, maybe soon two options to achieve his goal of rerunning flaky tests in JUnit 5 - no need for native support. :wink:
Related to "flaky tests": https://twitter.com/michaelbolton/status/1032125004480237568
He should sooooo change his name - https://www.youtube.com/watch?v=ADgS_vMGgzY
According to @anonygoose in https://github.com/junit-team/junit5/issues/1079#issuecomment-418351588 the Rerunner extension is "dangerously buggy, and inactive".
@anonygoose Could you please post an example that showcases its bugginess?
Hey Marc,
import io.github.artsok.RepeatedIfExceptionsTest;
public class RerunnerBugTest {
@RepeatedIfExceptionsTest(repeats = 3)
void test1() {
System.out.println("Running test 1");
}
@RepeatedIfExceptionsTest(repeats = 3)
void test2() throws Exception {
System.out.println("Running test 2");
throw new Exception("How does this not error? It never runs!");
}
@RepeatedIfExceptionsTest(repeats = 3)
void test3() {
System.out.println("Running test 3");
}
}
The above set of tests will run test1, and then skip test2 and test3. As soon as one test annotated with the annotation passes, it will skip any other tests annotated with it.
Uploaded the full example here: https://github.com/anonygoose/rerunner-bug-example
Hi guys :)
I can make Pull Request with ReRun flasky mechanism, but i need information :)
Getting official support for a retry feature would be great.
Ideally, I'd like to see something like a @RetryableTest annotation that allows one to specify:
- maximum number of retries
- configurable exceptions that initiate a retry, defaulting to {Throwable.class}
- potentially a minimum success rate
Each of these points are configurable on the current Rerunner extension, and are useful. Carrying them over to any official support seems sensible.
Further consideration may need giving to:
- Making retries work across the entire life-cycle, wherever the exception occurs (BeforeEach, AfterEach, etc)
- What the proper outcome of the test is. Current rerun implementations end up with a lot of tests being reported as skipped. This is a bit messy when trying to interpret results.
It would be nice to know whether the JUnit 5 team is considering implementing this.
- If so, I will not continue working on my experiments.
- If not, this issue can be closed and @artsok and I can decide how to move forward with his extension and my snippets.
@artsok: Would you consider integrating your extension into JUnit Pioneer? You would get more visibility and more hands to fix issues or implement features.
dangerously buggy, and inactive
I wanted to second this. I came to a similar conclusion a few weeks ago. I had to give up on other attempts, not feeling they were entirely safe, clean, light-weight, or stable. Instead, I ended up making my own copy of RepeatedTestExtension then made it use a custom TestTemplateInvocationContext Iterator with a TestExecutionExceptionHandler that conditionally throws a TestAbortedException. This seems to be the general approach I've seen floating around but at least I have my own light-weight solution that I have control over. I wish artsok and pioneer the best of luck but it would definitely be nice to have a clean, official, stable, and trusted solution.
Just to add my 2 cents to this issue. We also have such flaky tests and were using the Surefire rerunFailingTestsCount in order to circumvent this. However, since our migration to JUnit Jupiter we noticed that this does not work anymore 😄. It would be great if there is something official from JUnit or maybe the appropriate plugins (Maven, Gradle) that would still have this support.
If you ask me I think that it is a bit better if we can do this expressively in the test itself and not have it global for the entire build.
Team Decision: As a first step, we think this should be maintained externally as an extension based on @TestTemplate. We might potentially introduce support for rerunning flaky tests in core, but then it should support all kinds of testable methods, including @ParameterizedTests. We'd be happy to provide guidance and review such an extension, whether in the rerunner project, junit-pioneer or another project.
I've just pushed my proof of concept to Pioneer, I'd like to implement that feature there and am interested in ideas, feedback, and even code if someone wants to write it. :smile_cat:
@marcphilipp This can be closed, right?
Surefire can do this, but unfortunately only for JUnit 4: https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html I am not sure what the equivalent for Gradle is and if one of its plugins support retries.
What we need is that in the Surefire test result XML files the information about retries are contained. We use Allure for reporting, and I believe it processes these information in order to display how often a test has been repeated and what failures occured in each retry. Hopefully Surefile will soon implement this for JUnit 5, but I did not find any issue about that. I just created an issue for it.
A solution just for within an IDE will not help us at all. Think on CI/CD and the test result reporting.
I dislike the name ¨RepeatIfFailedTest¨, four words in a long camel case. Why not just @Retry?
My team and I are very anxious to upgrade to JUnit 5, but this has been the hang up for us. May I ask if there are any updates for this? I have noticed that the Rerunner project and Pioneer have yet to implement the ability of stacking a Retry annotation on top of @ParameterizedTests which is key to what we are wanting to accomplish.
@ekmaughan Thanks for letting us know that this feature is important for you. I'm afraid I don't have any updates for you at this point. I agree that this would be a useful feature for integration tests that depend on external resources.
@arcuri82 Have you submitted an issue to the Surefire project for this?
Hey @marcphilipp
One more vote but we would prefer configuration like the surefire rerunFailingTestsCount feature.
We have been using JUnit5 since its experimental days with great success. We are using it for integration testing of our API and like others, between network issues and platform issues, there is some level of unpredictability which is magnified because we are hitting our api so hard now that we can run things in parallel.
We have many tests and no one test is likely to be more "flaky" than another. The surefire rerunFailingTestsCount was actually the ideal solution for us rather than a special annotation per test. Ideally we would like to configure this kind of a feature for all tests.
I can see the use case for both a test specific annotation, or a configuration for all tests.
@dlanaghen Makes sense! We could add a global config parameter for the default.
Hey @ekmaughan
You can try to use my implementation with support @ParameterizedTests which I did today. As all main parameterized classes have default visibility I need to set mine owns. For that reason, I introduce a new annotation @ParameterizedRepeatedIfExceptionsTest.
Example:
private ThreadLocalRandom random = ThreadLocalRandom.current();
/**
* By default total repeats = 1 and minimum success = 1.
* If the test failed by this way start to repeat it by one time with one minimum success.
*
* This example with display name, repeated display name, 10 repeats and 2 minimum success with exceptions.
* Exception depends on random number generation.
*/
@DisplayName("User payment")
@ParameterizedRepeatedIfExceptionsTest(name = "payment amount was {0}",
repeatedName = " (Repeat {currentRepetition} of {totalRepetitions})",
repeats = 10, exceptions = RuntimeException.class, minSuccess = 2)
@ValueSource(ints = {40, 55, 61, 700})
void errorParameterizedTestWithDisplayNameAndRepeatedName(int argument) {
if (random.nextInt() % 2 == 0) {
throw new RuntimeException("Exception in Test " + argument);
}
}
IDEA Report:
Explaining: First test with argument '40' failed. Repeat this test with first argument. Wait the sitaution when will have 2 minimum success runs. Then go to the next test with second argument and run it. With second argument way we can see that it was no 2 minimum success follow each other. Try to repeat 10 times but wihout result, go next test with argument '61' and so on.
Allure Report (another test run):
Checked with this Junit version
<junit.jupiter.version>5.4.2</junit.jupiter.version>
<junit.platform.version>1.4.2</junit.platform.version>
You can try this extension:
<dependency>
<groupId>io.github.artsok</groupId>
<artifactId>rerunner-jupiter</artifactId>
<version>2.0.1</version>
<scope>test</scope>
</dependency>
The main logic you can find in ParameterizedRepeatedTestExtension.class. Source code you can find here. Don't forget to send star if u want.
Thanks @artsok. Looking forward to when you’ve made sure it works parallelized too!
Hi, Any progress on this ? We rely on the 'rerunFailingTestsCount' feature of maven-surefire-plugin and I just learned the hard way that it does not work with JUnit5.
@toby1984 While we'd still like to implement this, we haven't had time to look into it more closely, yet.
One open question is if/how we should report repetitions with "allowed failures". @artsok seems to have reported them as failed, I could also envision reporting them as aborted. Or we could not report them at all and only fail the test in the end if it didn't pass after the configured number of retries.
Another open question is the lifecycle of a repetition. Spock allows to configure which parts of the setup get executed for a retry. I think we can make it simple to start with and repeat the whole lifecycle like we do for repeated and parameterized tests.
Thoughts?