Re-running failed tests with maven surefire does not work

[mimimizinova]

I'm trying to use surefire feature -Dsurefire.rerunFailingTestsCount on failing test. If count=1, it's supposed to be run again one time, but it doesn't:

Running Test3 Tests run: 2, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 0.055 sec <<< FAILURE! - in Test3 [0] AAA,1 (testTestTest)(Test3) Time elapsed: 0.005 sec <<< FAILURE! java.lang.AssertionError at Test3.testTestTest(Test3.java:55)

initializationError(org.junit.runner.manipulation.Filter) Time elapsed: 0 sec <<< ERROR! java.lang.Exception: No tests found matching Method [0] AAA,1 (testTestTest)(Test3) from org.junit.internal.requests.ClassRequest@27ddd392

Results :

Failed tests: Test3.testTestTest:55 Tests in error: Filter.initializationError » No tests found matching Method [0] AAA,1 (testTe...

Tests run: 2, Failures: 1, Errors: 1, Skipped: 0

I found bugreport for surefire which was fixed in 2.19. I took sample project from this report and run it as-is with surefire 2.19.1, junit 4.12 (it worked), then replaced default runner with JUnitParamsRunner, @Parameters annotation (to fit replaced runner) and got an error.

import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import junitparams.JUnitParamsRunner;
import junitparams.Parameters;

@RunWith(JUnitParamsRunner.class)
public class Test3 {

    private static boolean test;

    static {
        test = true;
    }

    @Test
    @Parameters({"AAA,1"})
    public void testTestTest(String p1, int p2) {
        boolean current = test;
        test = !test;
        Assert.assertFalse(current);
    }
}

I also tried @TestCaseName("{method}[{index}]") annotation, but it didn't help. Could it be a bug in runner?

[zbychupragmatists]

It seems that this is problem in our library. I gave a quick look at it and reruning tests is implemented by applying filters on test suite. Filter takes all failed test methods names as condition for filtering. Problem is that when using parametrized tests description of test methods are generated "on the fly" and when applying filtering junit cannot find any tests to run e.g. I created simple failing test

@RunWith(JUnitParamsRunner.class)
public class FailingTest {

    @Test
    @Parameters({"abc,3", "defg,5"})
    public void hasLength(String text, Integer length) {
        assertThat(text).hasSize(length);
    }
}

And ran it with mvn -Dsurefire.rerunFailingTestsCount=2 test it fails for second parameters set "defg,5". So junit is trying to rerun it by applying filter to suite with failed method name description. Problem is that this method has name hasLength(defg,5) and after applying such filter junit throws exception Run 1: Filter.initializationError » No tests found matching Method hasLength(defg,5).... It occurs cause with current implementation of junitparams we cannot filter single execution of set of parameters (in example "defg,5").

This issue is related to https://github.com/Pragmatists/JUnitParams/issues/122 . I think in near future we will solve it. We have some idea how to implemented it but we need to agree on goal of https://github.com/Pragmatists/JUnitParams/issues/122.