Priority of test methods is ignored

[AndiCover]

TestNG Version

org.testng:testng:7.0.0

(This worked just fine with version 6.9.6)

Expected behavior

Because of the set priorities I expect following execution order: test05 test04 test02 test03 test01

This works when running from Eclipse IDE.

Actual behavior

Unfortunately, when running from IntelliJ or Gradle (tried with 5.4.1 and 5.6.2) it results in following execution order: test01 test02 test03 test04 test05

Is the issue reproductible on runner?

• [ ] Shell
• [ ] Maven
• [x] Gradle
• [ ] Ant
• [ ] Eclipse
• [x] IntelliJ
• [ ] NetBeans

Test case sample

I have following test class:

public class TryOut {

	@Test(priority = 4)
	public void test01() {
		System.out.println("test01");
	}

	@Test(priority = 3)
	public void test02() {
		System.out.println("test02");
	}

	@Test(priority = 3)
	public void test03() {
		System.out.println("test03");
	}

	@Test(priority = 2)
	public void test04() {
		System.out.println("test04");
	}

	@Test(priority = 1)
	public void test05() {
		System.out.println("test05");
	}
}

build.gradle:

plugins {
    id 'java'
}

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.testng:testng:7.0.0'
}

// Test Logging
test {
	useTestNG()
    testLogging {
        showStandardStreams = true
    }
}

The Gradle command: ./gradlew test --tests mypackage.TryOut

Here the complete Gradle project: TestNgExample.zip

[AndiCover]

Added a possible workaround to the related stackoverflow question: https://stackoverflow.com/questions/55125096/gradle-ignores-priority-in-testng

[krmahadevan]

@AndiCover - I am not able to reproduce this error using TestNG 7.0.0 (Your stack overflow post refers to you using a beta version and am referring to the latest released version )

https://mvnrepository.com/artifact/org.testng/testng/7.0.0

Here's the output when running this from maven

[INFO] --- maven-surefire-plugin:3.0.0-M1:test (default-test) @ testbed ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.rationaleemotions.github.issue2170.TryOut
test05
test04
test02
test03
test01
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.214 s - in com.rationaleemotions.github.issue2170.TryOut
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:32 min
[INFO] Finished at: 2019-10-07T19:24:09+05:30
[INFO] ------------------------------------------------------------------------

And here's the output from IntelliJ

[krmahadevan]

Closing this issue. Please comment if this is still a problem with 7.0.0 (Not the beta versions)

[AndiCover]

The stackoverflow question comes not from me. I just answered it and provided a workaround.

I am using the released version 7.0.0

[krmahadevan]

@AndiCover - Ah ok.. My bad. But I can't seem to be able to reproduce the problem using 7.0.0. Do you mind sharing a simple standalone project (as a gradle/maven project that I can use to recreate this issue) ?

[AndiCover]

I updated the description and added a example project.

When I run the tests from command line I get following output:

[krmahadevan]

@AndiCover - I think Gradle is perhaps doing something extra here...

Just to test out my theory i created a suite xml file which looks like below

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >
<suite name="Smoke Test" verbose="2">
  <test name="Smoke Test" verbose="2">
    <classes>
      <class name="mypackage.TryOut"/>
    </classes>
  </test>
</suite>

altered the gradle file to look like below

test {
	useTestNG() {
        suites "src/test/resources/suite.xml"
    }
    testLogging {
        showStandardStreams = true
    }
}

and then ran the tests, and this is what i see

22:20 $ ./gradlew test

> Task :test

Gradle Test Executor 3 STANDARD_OUT
    ...
    ... TestNG 7.0.0 by Cédric Beust ([email protected])
    ...


Smoke Test > Smoke Test > mypackage.TryOut > test05 STANDARD_OUT
    test05

Smoke Test > Smoke Test > mypackage.TryOut > test04 STANDARD_OUT
    test04

Smoke Test > Smoke Test > mypackage.TryOut > test02 STANDARD_OUT
    test02

Smoke Test > Smoke Test > mypackage.TryOut > test03 STANDARD_OUT
    test03

Smoke Test > Smoke Test > mypackage.TryOut > test01 STANDARD_OUT
    test01

BUILD SUCCESSFUL in 1s
3 actionable tasks: 2 executed, 1 up-to-date

[AndiCover]

Could of course be a Gradle problem but when using an older TestNG version e.g. 6.9.6 this works just fine. We faced this issue when we upgraded to the latest version.

[AndiCover]

I just tried the versions between 6.9.6 and 7.0.0 and it only behaves differently with 7.0.0.

[krmahadevan]

@AndiCover - I don't know a lot about Gradle... so am not sure what's goofing it up.. coz it works fine with Maven and also with the IDE as well...

[krmahadevan]

@juherr - any suggestions ?

[DanVill29]

hi, im also able to replicate it maven project in this version 7.0.0. Works ok in previous version.

[juherr]

@krmahadevan I think you'll be able to reproduce the issue with CLI without suite too because all tools are using the same entry point. Let me know if you need more help.

@DanVill29 Which Maven version? Which surefire version?

[DanVill29]

@juherr , I used maven version: 3.6.0

[krmahadevan]

@DanVill29 - Can you please share a sample project which we can use to reproduce this issue ?

[krmahadevan]

@juherr - No I am not able to reproduce this. I didn't try CLI but tried executing the tests directly via maven (see here and its working as intended )

[juherr]

I'm able to reproduce the issue in a fresh IntelliJ project but not if I add the test in the TestNG project itself. I confirm the issue happens only with 7.x versions and not previous ones.

[gdom12345]

I'm also seeing this with 7.x. Only happening when I try to run a Suite of Suites, via testng plugin in IntelliJ or when run through maven targeting multiple suites. Running a single suite does work.

[krmahadevan]

@gdom12345 - Can you please share a reproducible sample project that can be used?

[Kanaduchi]

Problem still exists

I can easily reproduce it with testng 7.4.0. I run this test from intellij idea and Java 11. May be reason can be in Java 11 JDK:

public class TestNG {

    @Test(priority = 1)
    public void first4()
    {
        System.out.println("4");
    }

    @Test(priority = 2)
    public void first3()
    {
        System.out.println("3");
    }

    @Test(priority = 3)
    public void first2()
    {
        System.out.println("2");
    }

    @Test(priority = 4)
    public void first1()
    {
        System.out.println("1");
    }
}

[mayanktoc]

Problem with 7.4.0 is still exist. I am using Gradle project.

[AlexMirroff]

@gdom12345 I'm reproducing issue with TestNG 7.4.0 when I starts class src/test/java/TestNG.java in IDE in my project https://github.com/AlexMirroff/HW/blob/master/src/test/java/TestNG.java

in 6.14.3 is OK

[ispringer]

I see the issue with 7.3.0. I see it both when running the test via Maven/Failsafe and when running it via IntelliJ. I am using Java 11.

[Pratish89]

I am also able to reproduce the issue with 7.0.0--7.4.0 Using java. ivy+ant

[ispringer]

Any updates on this issue? It's preventing my team's project from upgrading to testng 7.x, which contains fixes for other bugs that are affecting us.

Thanks!

[krmahadevan]

@ispringer - I still have not been able to reproduce this problem using TestNG 7.5

I am attaching a sample project which contains a README.md that details the ways of running the test.

Can you please help take a look at the sample and see if it matches with what your test looks like. Please feel free to alter it so that it can be used to reproduce the problem. I can work on figuring out what is going wrong and fix it, if there's a problem. TestNgExample.zip