testng icon indicating copy to clipboard operation
testng copied to clipboard
verified publisher icon testng-team

BUG: Parameter alwaysRun=true for before-methods forces execution of those methods

Open kool79 opened this issue 8 years ago • 16 comments

TestNG Version

6.12, 6.11, ...

Expected behavior

According to official documentation parameter 'alwaysRun' for before-methods is used only to enable/disable filtering by groups. It must not force method execution if higher-level configuration(s) failed. In any case alwaysRun should not be used both for filtering and for execution forcing

Actual behavior

BeforeTest/BeforeClass/BeforeMethod with (alwaysRun = true) is executed when higher - level configuration failed (BeforeSuite/BeforeTest/BeforeClass)

Is the issue reproductible on runner?

  • [ ] Shell
  • [ ] Maven
  • [ ] Gradle
  • [ ] Ant
  • [ ] Eclipse
  • [x] IntelliJ
  • [ ] NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

public class AlwaysRunTests {

	@BeforeSuite
	public void failedBeforeSuite() {
		System.out.println("I'm beforeSuite. I should be alone in console log. Going to fail...");
		throw new RuntimeException();
	}

	@BeforeTest(alwaysRun = true)
	public void beforeTest() {
		System.out.println("BUG: I'm beforeTest and I was invoked (((( ");
		throw new RuntimeException();
	}

	@BeforeClass(alwaysRun = true)
	public void beforeClass() {
		System.out.println("BUG: I'm beforeClass and I was invoked (((( ");
		throw new RuntimeException();
	}

	@BeforeMethod(alwaysRun = true)
	public void beforeMethod() {
		System.out.println("BUG: I'm beforeMethod and I was invoked (((( ");
		throw new RuntimeException();
	}

	@Test
	public void testMethod() {
		System.out.println("I'm testMethod");
	}
}

Console output (stacktraces truncated):

I'm beforeSuite. I should be alone in console log. Going to fail...

java.lang.RuntimeException
	at qa.ok.testng.AlwaysRunTests.failedBeforeSuite(AlwaysRunTests.java:14)

BUG: I'm beforeTest and I was invoked (((( 

java.lang.RuntimeException
	at qa.ok.testng.AlwaysRunTests.beforeTest(AlwaysRunTests.java:21)

BUG: I'm beforeClass and I was invoked (((( 

java.lang.RuntimeException
	at qa.ok.testng.AlwaysRunTests.beforeClass(AlwaysRunTests.java:28)

BUG: I'm beforeMethod and I was invoked (((( 

java.lang.RuntimeException
	at qa.ok.testng.AlwaysRunTests.beforeMethod(AlwaysRunTests.java:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:523)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:224)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:599)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:877)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1201)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:776)
	at org.testng.TestRunner.run(TestRunner.java:634)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:425)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:420)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:385)
	at org.testng.SuiteRunner.run(SuiteRunner.java:334)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1318)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1243)
	at org.testng.TestNG.runSuites(TestNG.java:1161)
	at org.testng.TestNG.run(TestNG.java:1129)
	at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

Test ignored.
===============================================
Default Suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 4, Skips: 0
===============================================

kool79 avatar Nov 21 '17 20:11 kool79

Thanks for the report.

How do you run the class?

Then, there are 2 interpretations:

  • no group means the default no-group, and the behavior is the expected one (before the no-group)
  • no group is the default group, and the current behavior is an issue.

Related to #1574 Ping @krmahadevan

juherr avatar Nov 27 '17 10:11 juherr

no group is no group. I run this without any groups defined, with default IntelliJ IDEA runner both from context menu on test-method and from context menu on testng.xml:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="my-suite">
	<test name ="my-test">
		<packages>
			<package name="qa.ok.testng.*"/>
		</packages>
	</test>
</suite>

(package qa.ok.testng.* contains only one class "AlwaysRunTests" listed in my prev. comment) No other settings/listeners/etc are present. I've attached zip with pom, java and testng.xml: always-run-test.zip

kool79 avatar Nov 27 '17 14:11 kool79

I reword:

no group is the default group

Your current suite will run all tests in the package, right? And not just tests without group.

BTW, it is debatable and we can only quick fix regression for the moment.

juherr avatar Nov 27 '17 15:11 juherr

Your current suite will run all tests in the package, right?

Yes. I understand. You mean my tests-list can also have some tests with groups so this can be the reason why Before-methods invoked. But I don't want to run those methods if any of the 'parent' configurations failed. In any case current behavior does not corresponds to documentation. I don't know if it is the documentation bug or it the bug in code which was introduced after documentation was published. In documentation the parameter "alwaysRun" has 2 different meanings for before- and after- methods :

  • For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to.
  • For after methods (afterSuite, afterClass, ...): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.

So first is about groups (always run regardless on group) and second - about failed/skipped (alwaysRun regardless on failed/skipped). But currently 'alwaysRun' for before-methods includes behavior for after-methods: when set to true:

  1. configuration method will be run even if one or more methods invoked previously failed or was skipped AND
  2. configuration method will be run regardless of what groups it belongs to.

kool79 avatar Nov 27 '17 17:11 kool79

I've found this bug was introduced in testng-6.9.5 when fixing #420 Below the log for v6.9.4 (note that no other before methods were invoked except before-suite):

  D:\dev\AC\git\test-testng\testng.xml
I'm beforeSuite. I should be alone in console log. Going to fail...

java.lang.RuntimeException
	at qa.ok.testng.AlwaysRunTests.failedBeforeSuite(AlwaysRunTests.java:14)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:517)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:140)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:296)
	at org.testng.SuiteRunner.run(SuiteRunner.java:259)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1199)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1124)
	at org.testng.TestNG.run(TestNG.java:1032)
	at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
Test ignored.
===============================================
my-suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 1, Skips: 3
===============================================

kool79 avatar Nov 27 '17 17:11 kool79

Ok, thanks for digging :+1:

juherr avatar Nov 27 '17 19:11 juherr

@juherr - I am not sure how to go about fixing this because :

I am not able to figure out how to determine if the user ran with group filtering or if the user ran tests as is. The reason I say this is because a user may have applied a method selector (either as implementation or as a beanshell expression) to filter methods.

So if I cannot determine if the tests are being run by applying groups, I will not be able to determine if alwaysRun=true should be considered or not.

krmahadevan avatar Nov 29 '17 03:11 krmahadevan

Guys, sorry but I just want to prevent breaking changes in testng, so will try to explain my opinion: Before testng v6.9.5 parameter alwaysRun had very different meaning in @Before- and @After- methods. Below I tried to describe behavior for that version (based on official testng documentation, javadocs and implementation):

  • In context of @BeforeSuite, @BeforeTest, @BeforeClass, @BeforeMethod : schedule this method for execution regardless to which group it belongs / does not belong (of cause if corresponding @Tests were scheduled) Please note: (by implementation up to 6.9.5): -- method will be skipped if any parent or preceding configurations failed. -- this behavior does not depend on config-failure-policy

  • In context of @AfterSuite, @AfterTest, @AfterClass, @AfterMethod : (from javadoc and official documentation): configuration method will be run even if one or more methods invoked previously failed or was skipped

  • In context of @Test: (from javadoc): run test-method even if it depends on a method that failed. This attribute ignored if test doesn't depend on any method or group. Please note: -- test-method will be skipped (will not run) if it depends on skipped methods. -- test-method will be skipped in every case when any of parent configurations failed.

Last note is very impotent. It means: even when if you force to execute @BeforeMethod after @BeforeClass was failed , all the methods which belongs to class will be skipped. So why we need to force execution of @BeforeMethod if test ignored in any way???

I think this behavior ( v6.9.5) should be saved because it was well-designed by @cbeust and it have meaning. There are no reasonable situations in testing when you should continue to execute 'child' prerequisites after 'parent' failed.

@krmahadevan, If I understand properly from PR 9d5e6d7, you want to force execution for before-methods only when configfailurepolicy = continue. Please use SRP principle for parameters in testng. Currently (v13.1) configfailurepolicy used to avoid multiple execution of the same configuration method. (Same means: method with concrete signature which belongs to concrete instance) Please don't use this parameter for any other purpose. This will break compatibility and will make process of testng configuration ugly.

Finally: IMHO to fix current issue you should revert changes which were made in scope of #420 (328620f) - it was wrong fix. Maybe it fixed behavior in particular case but it does not fix root cause. I'm almost sure that issue #420 can be reproduced with or without alwaysRun on @BeforeSuite

kool79 avatar Dec 06 '17 21:12 kool79

@kool79 It looks you have a good knowledge of the issue. Maybe you can propose a pull request with the fix?

juherr avatar Dec 06 '17 21:12 juherr

Sorry.. by mistake clicked the wrong button.

krmahadevan avatar Dec 07 '17 02:12 krmahadevan

@kool79 Just want to "upvote" this issue: I agree with your assessment, and find the current behavior frustrating. Given your extensive knowledge of the issue, would you mind trying to submit a fix? (i.e., reverting #420 but also proposing a better fix for that?)

ntulele avatar Mar 13 '18 18:03 ntulele

+1, our team affected by this issue. Is there any work-around to force skip of before/after methods by manipulation with IContext for example?

aTan-aka-Xellos avatar Aug 29 '18 14:08 aTan-aka-Xellos

any workaround for this issue?

hg0605 avatar Oct 29 '18 09:10 hg0605

@hg0605 - No there aren't any workarounds for this issue at the moment.

krmahadevan avatar Oct 29 '18 13:10 krmahadevan

Any update on this? We want to terminate the suite at the BeforeSuite level, but are noticing all the beforeMethods are still running.

cnealaccela avatar Oct 18 '19 15:10 cnealaccela

I just ran into this too. I have the exact same use case where I have a @BeforeMethod that I want to run for all methods regardless of the group so I thought that using alwaysRun = true would be the solution. However, I want it to be still be skipped if the @Test is going to be skipped due to a failure like an @BeforeSuite method failing. Is there any work around for this?

bdrx312 avatar Jul 31 '24 03:07 bdrx312