Parallelism option ignored when using own ForkJoinPool

[fredericOfTestfabrik]

I'm using Junit 5.4.1 and am experiencing a weird issue related to parallelism.

I have a set of Tests that I want to execute in parallel. 4 of these tests are tagged with the tag "main". Which I want to execute using maven. However I only want to execute 2 of them at the same time. Thus I configured jUnit 5 as follows:

junit.jupiter.execution.parallel.enabled=true junit.jupiter.execution.parallel.config.strategy=fixed junit.jupiter.execution.parallel.config.fixed.parallelism=2

This config works when I invoke the following mvn command:

mvn test

However, when i add the parameter to only execute the Tests tagged with main:

mvn test -Dgroups=main

ALL of the tests (meaning 4) tagged with "main" are executed in parallel, when only 2 should be executed at the same time. It seems that somehow, the parallelism count is ignored. Unfortunately I can't verify which setting is used by JUnit5 or if this bug is related to mvn surefire or Junit5

Steps to reproduce

• Create a Test Suite with 5 or more Tests
• Tag 4 of them with a tag (e.g main)
• invoke mvn test -Dgroups=main

Context

• Used versions (Jupiter/Vintage/Platform): 5.4.1 Jupiter / no vintage / 1.4.1 platform
• Build Tool/IDE: maven 3.6 surefire 2.22.1

[marcphilipp]

Are the 4 tests all in the same test class or different test classes?

Could you please provide a small sample project?

[fredericOfTestfabrik]

Yes all 4 tests are in the same classes.

As I can't share the original test code I wrote a small sample snippet which tried to reproduce the problem. This is when I found the actual problem. This isn't related to tags but more to concurrency and the meaning of the junit.jupiter.execution.parallel.config.fixed.parallelism parameter. I try to explain:

  @Test @Tag("main")
    void done1() throws InterruptedException {

        System.out.println("BEGIN1");
        System.out.println(Thread.currentThread().getName());
        ForkJoinPool customThreadPool = new ForkJoinPool(4);
        RecursiveAction a = new RecursiveAction() {
            @Override
            protected void compute() {
                while (true) {
                    try {
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                    System.out.println(Thread.currentThread().getName());}
            }
        };
        customThreadPool.invoke(a);

        System.out.println("DONE1");


    }

    @Test @Tag("main")
    void done2() throws InterruptedException {
        System.out.println("BEGIN2");
        System.out.println(Thread.currentThread().getName());
        String name = Thread.currentThread().getName();
        threadNames.add(name);
        ForkJoinPool customThreadPool = new ForkJoinPool(4);
        RecursiveAction a = new RecursiveAction() {
            @Override
            protected void compute() {
                while (true) {
                    try {

When setting junit.jupiter.execution.parallel.config.fixed.parallelism=1 my expectation was that the second test would never start to execute, since the task in the custom thread pool does never return a result and thus the calling thread executing the test would block forever. In reality this happens:

---------- Run Test-Before ----------
BEGIN2
ForkJoinPool-1-worker-1

---------- Run Test-Before ----------
BEGIN1
ForkJoinPool-1-worker-0
ForkJoinPool-2-worker-1
ForkJoinPool-3-worker-1

My knowledge about ForkJoinPools is unfortunately limited as I didn't work with them before but it seems that when worker-1 is forced to wait on the result of the invoke, another worker (worker-0) is spawned to start with the second test. This essentially means that all tests in my class would run in parallel. So i guess the junit.jupiter.execution.parallel.config.fixed.parallelism controls how many tests can be active at the same time, meaning that if fewer tests are active (because the executing Thread yields), JUnit spawns additional threads to keep the number of active tests at this maximum level?

If this is not what happens I would be happy if you could help me understand what exactly is going on here.

In any case this behavior surprised me as my intuition would be that the tests that are executed in parallel must actually complete before another test can be started. While I can see that there might be reasons why one does not want to do this, is there a way to enforce the behavior I expect (i.e that I can control how many tests are started in parallel) ?

[marcphilipp]

The idea behind ForkJoinPool's parallelism setting and by extension also JUnit's is trying to ensure that about n threads are doing work at all times. Thus, if one of the threads is about to be blocked, it spawns an additional one to ensure that work is still being done.

@fredericOfTestfabrik Do your actual tests also start operations that use ForkJoinPool in one way or another?

[fredericOfTestfabrik]

First of all thank you for your explanation.

Regarding your Question:

Yes the actual test create fork join pools to let long running operations execute in parallel.

My tests use selenium to test a website. The test code is built in such a way that it can be run in parallel for different browsers. Each test thus runs the selenium code in parallel for 2 different browsers. When I don't use junit5s parallelism, this runs fine.

Now want I want to do is to run multiple testcases at the same time. However the total amount of available browsers is limited. E.g I have 4 browsers available so I want to run 2 testcases at most at the same time, because each of the testcase again needs 2 browsers.

My guess is that at some point the testcode needs to yield (because it waits on a response or there is a thread sleep needed to ensure some stability). And this is when the additional tests are run. So what I need would be a setting or functionality to ensure that no more than 2 tests are run in parallel even if one test case is suspended for a certain amount of time.

Edit: I'm sorry for the mess. I am writing this from my phone and accidentally hit the close button

[marcphilipp]

Thanks for the explanation! I'll take a closer look in the next few days.

[fredericOfTestfabrik]

Is there an update on this? Do you need additional Information?

[marcphilipp]

@fredericOfTestfabrik I should know better than to make promises like that, sorry! Unfortunately, I haven't had a chance to investigate further, yet.

[sormuras]

Created https://github.com/sormuras/junit5-1858-fixed-parallelism to investigate this issue.

Some notes:

• I could verify the described behaviour - spawning a custom ForkJoinPool inside a test enables other tests to be executed in parallel, exceeding the configured fixed maximum value.
• Using a dead-simple Thread.sleep(long) to simulate a long duration test execution works out perfectly.
• TODO Try an ExecutorService to simulate some concurrent work within a test method...

[sormuras]

Changed the title to reflect that the underlying issue doesn't need tags or Maven to show up.

[sormuras]

Using an ExecutorService service = Executors.newFixedThreadPool(4); within a test method works as expected.

Seems like multiple ForkJoinPool instances steal work from each other...

Does anybody know a way to prevent this? By tagging/marking tasks as belonging to a specific pool only?

[sormuras]

Using Stream#parallel() seems not to mess with ForkJoinPool, though...

[sormuras]

work-around

Using a custom strategy that also fixes the maximum (and core?) pool size to same value as the parallelism option, solves the issue described here.

The default FIXED strategy adds 256 to get the maximum pool size: https://github.com/junit-team/junit5/blob/5e41ebe612fd67e905f96f1dd3184a071b65be17/junit-platform-engine/src/main/java/org/junit/platform/engine/support/hierarchical/DefaultParallelExecutionConfigurationStrategy.java#L44

[marcphilipp]

IIRC the problem with that is that the ForkJoinPool sometimes spawns additional threads to reach the target parallelism level, e.g. when one is about to be blocked (see ManagedBlocker).

[marcphilipp]

@fredericOfTestfabrik Please give the workaround posted by @sormuras above a try and let us know if that works for you.

[fredericOfTestfabrik]

Sorry i was busy the last 2 weeks.

I gave the solution a try but unfortunately it did not work with the code I posted above.

I copied the strategy @sormuras posted, set the config.strategy to custom and set the custom.class to the Strategy class. Additionally, i set all configuration options inside the strategy to 1. Thus my expectation was that only 1 tests would start at all, as the test effectively enters an infinite loop. However both tests were started as before.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.

[stale[bot]]

This issue has been automatically closed due to inactivity. If you have a good use case for this feature, please feel free to reopen the issue.

[mythsunwind]

@marcphilipp I think this ticket should be reopened. I just ran into the issue with JUnit 7.5.2 while writing my own test engine and applying it to a big number of test cases. Once newFixedThreadPool was used in a test the maximum amount of parallel tests increased. The workaround is to use stream().parallel() as described by @sormuras . I was not able to use the workaround with a custom strategy. My custom strategy also had the same problem.

I think this issue should be fixed in JUnit because it is impossible to be sure that a test is not using a library that includes a newFixedThreadPool.

[mythsunwind]

Hi @marcphilipp Is there any chance to tackle this issue somehow? Selenium 4 is out now and unfortunately is using ExecutorService with newFixedThreadPool inside which results in JUnit 5 tests maxing out the maximum parallel pool size and not stopping new tests from running when it is exceeded. This fills up the queue on selenium grids until reaching 256 + your parallel count setting.

I know that the current documentation of JUnit 5 says "JUnit Jupiter does not guarantee that the number of concurrently executing tests will not exceed the configured parallelism" and "Thus, if you require such guarantees in a test class, please use your own means of controlling concurrency" but this will probably affect any person run into this whenever combining JUnit 5 with selenium-java >= 4 and trying to run tests in parallel.

[marcphilipp]

The only idea I have off the top of my head is to prevent the fork join pool for compensating one way or another but I haven't investigated the feasibility of this, yet.

@mythsunwind Would you be interested in digging deeper?

[christophs78]

I´m also fighting with this issue. (see https://github.com/SeleniumHQ/selenium/issues/10113, https://github.com/primefaces/primefaces/pull/8145) Implementing our own ParallelExecutionConfigurationStrategy only helps for Java 11, but does not work at all with Java 8 and has different issues with Java 17.

[christophs78]

Searching for ideas how to fix the Selenium 4 / JUnit 5 / Parallelism - issue....

• One option would be talking to Selenium-team about bringing back OkHttp as an alternative option to AsyncHttpClient. (Because we know this issue was introduced by Selenium switching from OkHttp to AsyncHttpClient.)
• Do we understand why this work´s with Java 11 but (due to different issues) not with Java 8 and Java 17? (One minor difference i found related to Java 8 is the way how JUnit 5 instanciates ForkJoinPool. But propably this does not matter for this issue. https://github.com/junit-team/junit5/blob/17ea54641fc8e3a116fc1de60abd44f9bebc8495/junit-platform-engine/src/main/java/org/junit/platform/engine/support/hierarchical/ForkJoinPoolHierarchicalTestExecutorService.java#L82)

[christophs78]

some debugging...

Settings

• parallelism = 2
• tests = 5
• custom impl. of ParallelExecutionConfiguration, ParallelExecutionConfigurationStrategy

Java 8

look at size and active

Java 11

Java 17

org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure.
Build info: version: '4.1.0', revision: '87802e897b'
System info: host: 'DESKTOP-6I7IEPJ', ip: '192.168.0.7', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '17.0.1'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [--headless], extensions: []}, goog:loggingPrefs: org.openqa.selenium.logging..., pageLoadStrategy: normal}], desiredCapabilities=Capabilities {browserName: chrome, goog:chromeOptions: {args: [--headless], extensions: []}, loggingPrefs: org.openqa.selenium.logging..., pageLoadStrategy: normal}}]

	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:577)
	at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:246)
	at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:168)
	at org.openqa.selenium.chromium.ChromiumDriver.<init>(ChromiumDriver.java:108)
	at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:104)
	at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:91)
	at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:80)
	at org.primefaces.selenium.internal.DefaultWebDriverAdapter.createWebDriver(DefaultWebDriverAdapter.java:93)
	at org.primefaces.selenium.spi.WebDriverProvider.get(WebDriverProvider.java:65)
	at org.primefaces.selenium.internal.junit.WebDriverExtension.beforeAll(WebDriverExtension.java:36)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$12(ClassBasedTestDescriptor.java:391)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:391)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:207)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:82)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:148)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:195)
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
	at java.base/java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: java.util.concurrent.RejectedExecutionException: Thread limit exceeded replacing blocked worker
	at java.base/java.util.concurrent.ForkJoinPool.tryCompensate(ForkJoinPool.java:1819)
	at java.base/java.util.concurrent.ForkJoinPool.compensatedBlock(ForkJoinPool.java:3446)
	at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3432)
	at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1939)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
	at org.openqa.selenium.remote.service.DriverService.start(DriverService.java:217)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:95)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:559)
	... 30 more

[titusfortner]

For what it's worth, selenium can't go back to OkHttp, because the new features in Selenium 4 require the asynchronous functionality from AsyncHttpClient

[christophs78]

IMO both sides (JUnit 5 and Selenium 4) do nothing "wrong" on their own. But together (paired with Parallelism) something goes wrong. My current best bet is the combination of ForkJoinPool (used by JUnit 5) with Executors.newFixedThreadPool (used by Selenium 4 as part of https://github.com/SeleniumHQ/selenium/blob/28070d2de242978beff7f5ffa0c54fbe805df401/java/src/org/openqa/selenium/remote/service/DriverService.java#L67) causes this trouble. Does this sound reasonable?

I already checked out JUnit 5 sources and will play around with an alternative implementation for org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService. Maybe based on ThreadPoolExecutor. Just to figure out whether this is an reasonable option.

Update: After doing some testing and research (see attached commit) i´m pretty sure my "best bet" is the reason for this issue. My ThreadPoolHierarchicalTestExecutorService draft-implementation work´s (in it´s very early and rough stage) fine with Java 8, 11 and 17.

[christophs78]

With the second (cleaned up and forced pushed) commit we are getting into shape. It even work´s fine on Java 8, 11 and 17 even without of implementing our own ParallelExecutionConfigurationStrategy.

[christophs78]

Maybe there´s a ForkJoinPool-regression in Java 17 causing this issues. We are further investigating. (https://github.com/SeleniumHQ/selenium/issues/10113#issuecomment-1003525910)

[christophs78]

Java 17 - issue - reproducer: https://github.com/christophs78/ForkJoinPoolReproducer Any idea how to approach the OpenJDK-team with this? https://bugs.openjdk.java.net/secure/Dashboard.jspa is readonly (https://dev.karakun.com/rico/2021/02/22/openjdk-bug-tracker.html -> https://bugreport.java.com/bugreport/)

[christophs78]

There´s already https://github.com/junit-team/junit5/issues/2787 and https://github.com/junit-team/junit5/pull/2792 covering this. -> We just have to wait for JUnit 5.9.

[jmrg82]

We have been waiting for the final 5.9.0 version for a long time with many hopes, but this problem is still present for us :(

maven.compiler.source -> 1.8 junit-jupiter.version -> 5.9.0 selenium-java.version -> 4.3.0 selenium-jupiter.version -> 4.2.0 maven-surefire-plugin.version -> 3.0.0-M5

<configurationParameters>
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = concurrent
junit.jupiter.execution.parallel.config.strategy = fixed
junit.jupiter.execution.parallel.config.fixed.parallelism = ${threads}
</configurationParameters>

mvn clean test -Dtest=Class1Test,Class2Test,Class3Test -Dthreads=2 -Denv=qa -Dbrowser=chrome

-> the value of threads is ignored and 3 (or the same number of test classes) different browsers are opened in parallel instead of the fixed number of browsers. With webdriver 3 and older versions of jUnit this worked perfectly...