All scenarios from same feature file are running in isolation even if only one has the @isolated tag

[fslev]

👓 What did you see?

While executing scenarios in parallel using cucumber-junit-platform-engine and "isolated" as a global exclusive resource, it seems that all scenarios from same feature file are executed in isolation, even though only one of them has the @isolated tag

✅ What did you expect to see?

Only the scenario with the @isolated tag should run isolated. The other scenarios from same feature should run in parallel.

📦 Which tool/library version are you using?

Cucumber Java 7.4.1

🔬 How could we reproduce it?

Steps to reproduce the behavior:

1.) Set the following configuration inside junit-platform.properties

cucumber.execution.parallel.config.strategy = fixed
cucumber.execution.parallel.config.fixed.parallelism=8
cucumber.execution.exclusive-resources.isolated.read-write=org.junit.platform.engine.support.hierarchical.ExclusiveResource.GLOBAL_KEY
cucumber.junit-platform.naming-strategy=long

2.) Create two feature files having each 3 scenarios 3.) Tag the second scenario from first feature file with @isolated 3.) Run all tests

Actual: Scenarios running in the same time: Feature 2 - Scenario 1 Feature 2 - Scenario 2 Feature 2 - Scenario 3

Scenarios running in serial: Feature 1 - Scenario 1 Feature 1 - Scenario 2 (tagged with @isolated) Feature 1 - Scenario 3

You can see the following timeline report:

I must admint that I can't say for sure if this a cucumber-junit-platform-engine issue, a Junit 5 issue or something in between.

You can reproduce the problem by running the following showcase:

[email protected]:fslev/cucumber-showcase.git

mvn clean test

[mpkorstanje]

While Cucumber assumes all scenarios and examples are completely independent from each other and the feature file they're in JUnit 5 makes some more pessimistic assumptions about its containers.

From: https://github.com/junit-team/junit5/issues/2038#issuecomment-541171707:

The (pessimistic) assumption behind forcing children of containers with resource locks to use execution mode SAME_THREAD is that the resource might not be safe to access in e.g. @BeforeAll methods and in different threads from @Test methods. Thus it's not the same as having @ResourceLock on every @Test method. IIRC we had a similar issue for @Execution only applying to methods while being declared on the class level.

And: https://github.com/junit-team/junit5/issues/2038#issuecomment-675006959

I had discussed with @marcphilipp that there should be something like @ResourceLockChildren that only applies to the children of a TestDescriptor but not the descriptor itself, i.e. @ResourceLockChildren on the test class would be the same as applying @ResourceLock on each test method.

This has been some what alleviated by https://github.com/junit-team/junit5/issues/2423 but it would appear that conceptually global locks are still propagated upward and there is no ResourceLockChildren implementation yet:

https://github.com/junit-team/junit5/blob/5fdb138924f59703a6a4fcd0cdcde6d1a224a950/junit-platform-engine/src/main/java/org/junit/platform/engine/support/hierarchical/NodeTreeWalker.java#L73-L76

if (!globalLockDescriptor.equals(testDescriptor) && allResources.contains(GLOBAL_READ_WRITE)) {
	forceDescendantExecutionModeRecursively(advisor, globalLockDescriptor);
	advisor.useResourceLock(globalLockDescriptor, globalReadWriteLock);
}

[mpkorstanje]

@fslev looks like you did not push your changes to cucumber-showcase.

[fslev]

Sorry for that. They are pushed now. So it seems this is a Junit 5 constraint... In which case, maybe the documenation might be adjusted a little bit. That way, one would know to make a separate Feature file for specific @isolated scenarios.

[mpkorstanje]

Nah. I believe this is something that should be fixed in JUnit 5. Currently the JUnit Platform implements an assumption about one test engine (JUnit Jupiter) that does not apply to another (Cucumber).

[mpkorstanje]

Not much we can do here on the Cucumber side.