When BeforeMethod throw exception, the Test(With Retry Listener and driven by dataProvider) does not skip

[Li-Vincent]

TestNG Version

v7.4.0

Note: only the latest version is supported

Expected behavior

skip test When BeforeMethod throw exception

Actual behavior

When BeforeMethod throw exception, the Test(With Retry Listener and driven by dataProvider) does not skip

Is the issue reproductible on runner?

• [x] Maven
• [x] IntelliJ

Test case sample

Please, share the test case (as small as possible) which shows the issue

Test Scripts:

    package testScripts.sample.dataProvider;

    import listener.DefaultRetryAnalyzer;
    import org.testng.annotations.BeforeMethod;
    import org.testng.annotations.DataProvider;
    import org.testng.annotations.Test;
    
    public class DataProviderFailTest {
        @DataProvider(name = "arrayProvider")
        public Object[][] dataArray() {
            return new Object[][]{
                  {"Scenario_pass", 1.0d},
                  {"Scenario_fail", 2.0d},
                  {"Scenario_skip", 3.0d}
          };
        }

    @BeforeMethod()
    public void preCondition(Object[] params) {
        String scenarioName = (String) params[0];
        System.out.println("preCondition scenarioName is " + scenarioName);
        if (scenarioName.equals("Scenario_skip")) {
            throw new IllegalStateException("Config failed.", new UnsupportedOperationException());
        }
    }

    @Test(dataProvider = "arrayProvider", retryAnalyzer = DefaultRetryAnalyzer.class)
    public void testProvider(String scenarioName, double data) {
        System.out.println("scenarioName: " + scenarioName);
        System.out.println("data: " + data);
        if (scenarioName == "Scenario_fail") {
            throw new IllegalStateException("This Scenario should be failed.", new UnsupportedOperationException());
        }
    }
    }

RetryAnalyzer.java

    package listener;

    import org.testng.IRetryAnalyzer;
    import org.testng.ITestResult;

    public class DefaultRetryAnalyzer implements IRetryAnalyzer {
        private int retryCount = 1;
        private static final int maxRetryCount = 3; // max retry times

    @Override
    synchronized public boolean retry(ITestResult result) {
        if (retryCount < maxRetryCount) {
            retryCount++;
            return true;
        }
        return false;
    }
    }

TestXML set configfailurepolicy="continue"

[Li-Vincent]

Run Results:

[Li-Vincent]

If I remove retryAnalyzer = DefaultRetryAnalyzer.class in @Test()

the result is ok.

[krmahadevan]

@Li-Vincent - Here's what the documentation states in terms of configfailurepolicy from here

Whether TestNG should continue to execute the remaining tests in the suite or skip them if an @Before* method fails. Default behavior is skip.

Since your suite file is configured to continue, TestNG will ignore the @Beforex method results and continue to execute the Tests.

TestNG is working as designed here.

[Li-Vincent]

There is a question here. Skip will be ignored only when RetryAnalyzer.class is added, but skip will not be ignored even if configfailurepolicy = continue when RetryAnalyzer is not used.

[juherr]

@krmahadevan It looks there is an undocumented side-effect on skipped status (or configfailurepolicy) when a retry analyzer is used. I think using a retry analyzer should not have impacts on the result (except some retrials) Thoughts?

[krmahadevan]

@krmahadevan It looks there is an undocumented side-effect on skipped status (or configfailurepolicy) when a retry analyzer is used. I think using a retry analyzer should not have impacts on the result (except some retrials) Thoughts?

I am not sure @juherr Been super swamped at work and thanks to lockdowns again, juggling as a cook and as a maid as well :)

Haven't found time to look at this. I will squeeze in sometime over this weekend take this up and come back to you. Apologies for the delay in responding.

[thedsv]

@krmahadevan @krmahadevan any updates? I'm facing the same problem on v7.11.0

When using the Retry Listener:

• if @BeforeMethod fails on the first test execution attempt, then @Test is skipped (expected behavior)
• if @BeforeMethod fails on the second (or any subsequent) test execution attempt, then @Test is executed (incorrect behavior)