Test classes are being skipped

[valeriM1]

TestNG Version

7.6.1

Description

Not all of the test case classes are executed when the classes aren't ordered by alphabetical order, one class extends another class and the base class contains a test method which other tests in this class and the extended class depend on.

Expected behavior

All the test case classes should be executed.

Actual behavior

Only the first test case class is executed.

Is the issue reproducible on runner?

• [X] Shell
• [ ] Maven
• [ ] Gradle
• [ ] Ant
• [X] Eclipse
• [ ] IntelliJ
• [ ] NetBeans

Test case sample

<suite name="Suite"  >
  <test name="testCases" >
    <classes>
      <class name="tests.testCases.TestCaseB"/>
      <class name="tests.testCases.TestCaseA"/>
    </classes>
  </test>
</suite>

public class TestCaseA
{
    @Test
    public void testA1()
    {
        System.out.println("testA1");
    }

    @Test(dependsOnMethods = "testA1")
    public void testA2()
    {
        System.out.println("testA2");
    }
}

public class TestCaseB extends TestCaseA
{
    @Test(dependsOnMethods = "testA1")
    public void testB1() 
    {
        System.out.println("testB1");
    }
}

[krmahadevan]

I am not able to reproduce the problem. Please check github_2784.zip for a sample.

I have run this on the command line and from within IntelliJ. If this is a problem that is specific to running tests in eclipse, then please file the issue under https://github.com/cbeust/testng-eclipse

If this problem persists on the maven command line as well, please feel free to alter the attached project to be able to reproduce the problem and comment back.

[valeriM1]

I'm running TestNG programmatically as Java application and not via TestNg plugin.

package tests.testCases;

import java.util.ArrayList;

import org.testng.TestNG;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlTest;

public class test
{

    public static void main(String[] args) throws ClassNotFoundException
    {
        XmlSuite suite;
        XmlTest test;
        TestNG testNG;
        XmlClass xmlClass;
        ArrayList<XmlTest> tests;
        ArrayList<XmlClass> testClasses;
        ArrayList<XmlSuite> testSuites;

        testNG = new TestNG();
        testNG.setVerbose(10);

        testSuites = new ArrayList<XmlSuite>();
        suite = new XmlSuite();
        suite.setName("Suite");

        test = new XmlTest();
        test.setName("testCases");
        test.setSuite(suite);
        test.setGroupByInstances(true);

        testClasses = new ArrayList<XmlClass>();
        xmlClass = new XmlClass(Class.forName("tests.testCases.TestCaseB"));
        testClasses.add(xmlClass);
        xmlClass = new XmlClass(Class.forName("tests.testCases.TestCaseA"));
        testClasses.add(xmlClass);
        test.setClasses(testClasses);

        tests = new ArrayList<XmlTest>();
        tests.add(test);

        suite.setTests(tests);
        suite.setGroupByInstances(true);

        testSuites.add(suite);

        testNG.setXmlSuites(testSuites);

        testNG.run();
    }
}

[krmahadevan]

@valeriM1

I'm running TestNG programmatically as Java application and not via TestNg plugin.

You didn't mention this earlier, which is what caused the confusion. Thanks for clarifying this part now.

I am able to reproduce the problem with the below attached sample

  public void anotherTest() {
    TestNG testng = new TestNG();
    XmlSuite xmlsuite = new XmlSuite();
    xmlsuite.setName("my_suite");
//    xmlsuite.setGroupByInstances(true); <-- Enabling this gives another assertion failure
    XmlTest xmlTest = new XmlTest(xmlsuite);
    xmlTest.setName("my_test");
    xmlTest.setXmlClasses(Arrays.asList(
        new XmlClass(TestCaseB.class),
        new XmlClass(TestCaseA.class)
    ));
    System.err.println(xmlsuite.toXml());
    testng.setXmlSuites(Collections.singletonList(xmlsuite));
    MethodLogger listener = new MethodLogger();
    testng.addListener(listener);
    testng.run();
    List<String> expected = Arrays.asList(
        "com.rationaleemotions.TestCaseB.testA1",
        "com.rationaleemotions.TestCaseB.testA2",
        "com.rationaleemotions.TestCaseB.testB1",
        "com.rationaleemotions.TestCaseA.testA1",
        "com.rationaleemotions.TestCaseA.testA2"
    );
    assertEquals(listener.getLogs(), expected);
  }

There are two issues that I am noticing:

1. When xmlsuite.setGroupByInstances(true); is enabled, the number of methods executed itself is incorrect.
2. When the group by instances is not enabled, then the order of the method execution changes and the assertion fails.

[krmahadevan]

@valeriM1 - I spent some more time on this issue and it looks like TestNG is working as designed here.

When xmlsuite.setGroupByInstances(true); is enabled, the number of methods executed itself is incorrect.

The root cause of this problem is because by default preserve-order is always true and group-by-instances and preserve-order don't go well with each other especially when there's no factories involved (I believe group-by-instances is relevant ONLY when used along with @Factory produced test classes, because that's where there would be multiple test class instances of the same class). Flipping preserve-order to false would fix this.

When the group by instances is not enabled, then the order of the method execution changes and the assertion fails.

This is because the listener that I was using internally makes use of org.testng.ITestNGMethod#getQualifiedName which returns getRealClass().getName() + "." + getMethodName(). The sad part here is that, getRealClass().getName() is not the same as getInstance().getClass().getName() because getRealClass() refers to the parent class (when there's inheritance involved), but getInstance() always resorts to the current object's class. Long story short, if you flip getQualifiedName() with

testResult.getInstance().getClass().getCanonicalName() + "." + method.getTestMethod().getMethodName()

this failure would also go away.