Support parallel JUnit 5 tests

[cowwoc]

https://www.testcontainers.org/test_framework_integration/junit_5/ states:

Note: This extension has only be tested with sequential test execution. Using it with parallel test execution is unsupported and may have unintended side effects.

To me, the entire point of using Test Containers is to speed up parallel builds. I could easily get sequential tests working without it.

Please add support for parallel JUnit builds.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[cowwoc]

This issue is still relevant.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[cowwoc]

This issue is still relevant.

[bsideup]

@cowwoc have you considered using the Singleton Container pattern?

[cowwoc]

@bsideup This goes against what I am trying to do. I am trying to run multiple tests in parallel, each connecting to a separate docker container. Imagine each docker container running a database instance. I don't want one test influencing the data seen by another test.

[rnorth]

I agree we should look into this - the JUnit4 support allowed parallel containers from the beginning, but unfortunately this wasn’t as easy for JUnit5. We should look into this at some point, but to be honest I think our capacity means this will need to come from a community contribution.

_{Sent with GitHawk}

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[glockbender]

This issue is still relevant

[pixelstuermer]

Is there any activity on that topic?

[uraniumdawn]

This issue is still relevant.

[bsideup]

Contributions are welcome! :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[membersound]

@stale bot: I think this is still relevant.

[mfbieber]

@bsideup: @marciovmartins submitted this PR: https://github.com/testcontainers/testcontainers-java/pull/3220

Do you think this would safely enable parallelism? I would also appreciate much the possibility to run our test containers tests in parallel to drastically reduce our CI time!

[bsideup]

@mfbieber note that you already can do that, e.g. by making the container singleton: https://www.testcontainers.org/test_framework_integration/manual_lifecycle_control/#singleton-containers

[marciovmartins]

@bsideup this works with parallel tests in different classes? I think I had problems with that. I'll test again in the next days.

[bsideup]

@marciovmartins this works with _any_class in the current JVM (as it is a "standard" JVM's singleton, nothing fancy), no matter which test framework you're using.

[mfbieber]

Thanks, @bsideup for the quick reply!

I run into issues using this, though 😢. My tests start to fail when using the suggested singleton pattern.

TL;DR: By setting test.maxParallelForks(8) in the build.gradle file, the singleton pattern does not work for me. How would you suggest actually running test classes in parallel, so that the singleton pattern works?

I am running several test classes in parallel by setting test.maxParallelForks(8) in the build.gradle file (just to test the behavior). If I don't use the singleton pattern, but initialize the MockServerContainer in my parallel running test classes, all tests pass:

@Container
private final MockServerContainer mockServer = new MockServerContainer(
        DockerImageName.parse("jamesdbloom/mockserver:mockserver-5.10.0")
);

Several Containers are started of course (two, for two classes using testcontainers):

$ docker ps
CONTAINER ID   IMAGE                                      COMMAND                  CREATED          STATUS          PORTS                                            NAMES
ef086fc80f3a   jamesdbloom/mockserver:mockserver-5.10.0   "/opt/mockserver/run…"   21 seconds ago   Up 21 seconds   0.0.0.0:49297->1080/tcp                          competent_murdock
e5c798cf101b   jamesdbloom/mockserver:mockserver-5.10.0   "/opt/mockserver/run…"   45 seconds ago   Up 44 seconds   0.0.0.0:49291->1080/tcp                          exciting_thompson
55396df0884b   testcontainers/ryuk:0.3.0                  "/app"                   45 seconds ago   Up 45 seconds   0.0.0.0:49289->8080/tcp                          testcontainers-ryuk-7a681637-f378-4647-b694-5ab9a7d3bb12
56dccc28380d   testcontainers/ryuk:0.3.0                  "/app"                   45 seconds ago   Up 45 seconds   0.0.0.0:49288->8080/tcp                          testcontainers-ryuk-16d474eb-6c8c-4d89-b3ed-a498717da006

(And to my knowledge, there is no possibility to run several test methods of the same class in parallel in JUnit5. Someone, please correct me, if I am wrong 😸)

Now, with the singleton pattern, I have two separate test classes, that both extend the AbstractContainerBaseTest class:

import org.testcontainers.containers.MockServerContainer;
import org.testcontainers.utility.DockerImageName;

public abstract class AbstractContainerBaseTest {

    public static final MockServerContainer mockServer;

    static {
        mockServer = new MockServerContainer(
                DockerImageName.parse("jamesdbloom/mockserver:mockserver-5.10.0")
        );
        mockServer.start();
    }
}

Those JUnit5 test classes look typically something like this:

@ExtendWith(MockitoExtension.class)
@Testcontainers
class BetaGraphRepositoryTest extends AbstractContainerBaseTest {

    private Repository repository;

    @BeforeEach
    void setUp() {
        ServiceClient serviceClient= new ServiceClient(getMockServerUrl());
        repository= new Repository(serviceClient);
    }

    @Test
    void test_groups_returnsResultContainingException() {
        MockServerClient mockServerClient = new MockServerClient(mockServer.getHost(), mockServer.getServerPort());
        mockServerClient.when(request()
                .withPath("/groups"))
                .respond(response().withStatusCode(503));
    
        Optional<Result<GroupCollection, Exception>> groupsResult = repository.groups()
                .findFirst();
    
        assertThat(groupsResult).isPresent();
        assertThat(groupsResult.get().hasError()).isTrue();
        assertThat(groupsResult.get().getError()).isInstanceOf(RepositoryException.class);
    }
    
    @Nonnull
    private String getMockServerUrl() {
        return "http://" + mockServer.getContainerIpAddress() + ":" + mockServer.getServerPort() + "";
    }

This also starts two containers and the tests in the corresponding classes mostly fail:

$ docker ps
CONTAINER ID   IMAGE                                      COMMAND                  CREATED          STATUS          PORTS                                            NAMES
36dc4c357347   jamesdbloom/mockserver:mockserver-5.10.0   "/opt/mockserver/run…"   13 seconds ago   Up 12 seconds   0.0.0.0:49308->1080/tcp                          affectionate_mendeleev
9d5efcbc2bfd   jamesdbloom/mockserver:mockserver-5.10.0   "/opt/mockserver/run…"   14 seconds ago   Up 12 seconds   0.0.0.0:49307->1080/tcp                          tender_mcclintock
8d5f7f9d1e13   testcontainers/ryuk:0.3.0                  "/app"                   15 seconds ago   Up 14 seconds   0.0.0.0:49306->8080/tcp                          testcontainers-ryuk-d4c6a05a-d7ab-4e2e-aa1b-34b5bb464640
543c3857f26c   testcontainers/ryuk:0.3.0                  "/app"                   15 seconds ago   Up 14 seconds   0.0.0.0:49305->8080/tcp                          testcontainers-ryuk-7c63e720-a36a-4c43-9380-8c2598265e58

This only makes sense if the singleton class gets loaded twice. So this must be rather an issue with the Gradle forking mechanism and how that runs the test classes (but I am no expert with that). Since both test classes retrieve the container URL and port through the mockServer variable, they might be talking to the wrong container (making the tests fail)?

Also, I was wondering how the mock server knows what response to return if several tests try to access it at once (in the case the singleton pattern would work)?

It is interesting though, that the tests pass while not using the singleton pattern.

How would you suggest actually running test classes in parallel, so that the singleton pattern works?

[cowwoc]

@bsideup My original request was about running JUnit 5 tests in parallel. What do singleton test containers have to do with it? I don't want to reuse instances across tests. I want to run completely independent instances in parallel.

[bsideup]

@mfbieber if you fork the JVM, there is no way of sharing a static instance across the forks, that's a limitation of JVM, not Testcontainers. JUnit's parallel mode won't help either AFAIK.

You may try #1781, but you should understand that there will be no cleanup in this case

[bsideup]

@cowwoc parallel JUnit 5 tests run in the same JVM, this is why I suggested the singleton approach. If you fork the JVMs (e.g. with your build script), there is not much we can do.

[cowwoc]

@bsideup My goal is to run independent tests in parallel and ensure as much as possible that one test cannot impact others.

Consequently, while I don't really care whether I use a single or separate JVMs the latter is preferable because it guarantees that tests cannot impact one another. Regardless of whether we have a single or multiple JVMs, I need each test to run against a separate TestContainer instance.

[bsideup]

@cowwoc what you described already works out of the box. Containers started by Testcontainers are isolated from each other (random ports, temporary folders, etc etc)

[cowwoc]

@bsideup So you're saying we're able to run parallel JUnit 5 tests and this issue can be closed as fixed?

Can you comment on why https://www.testcontainers.org/test_framework_integration/junit_5/ still reads:

Note: This extension has only been tested with sequential test execution. Using it with parallel test execution is unsupported and may have unintended side effects.

Is this out of date or are we missing something?

[mfbieber]

@bsideup So you're saying we're able to run parallel JUnit 5 tests and this issue can be closed as fixed?

At least with my experiments, this seems to be working (but not completely smooth). Once we have a reliable setup, I would also be willing to contribute to at least updating/expanding the documentation. Here is what I did:

Concurrent execution via the JUnit5 configuration parameters (see https://junit.org/junit5/docs/snapshot/user-guide/#writing-tests-parallel-execution):

test {
    useJUnitPlatform {
        excludeTags 'xxx'
    }
    systemProperties = [
            'junit.jupiter.execution.parallel.enabled': true,
            'junit.jupiter.execution.parallel.mode.default': 'concurrent',
            'junit.jupiter.execution.parallel.mode.classes.default': 'concurrent'
    ]
}

And in the test classes:

@Nested
@Tag("integrationTests")
@Execution(CONCURRENT)
class MockServerTests { 

    @Container
    private final MockServerContainer mockServer = new MockServerContainer(
            DockerImageName.parse("jamesdbloom/mockserver:mockserver-5.10.0")
    );
    
   @BeforeEach
    void setUp() {
        mockServer.start();
    }
}

Without the mockServer.start() in the @BeforeEach, I was seeing this (only from time to time):

[ForkJoinPool-1-worker-15] INFO docker[jamesdbloom/mockserver:mockserver-5.10.0] - Creating container for image: jamesdbloom/mockserver:mockserver-5.10.0

Mapped port can only be obtained after the container is started
java.lang.IllegalStateException: Mapped port can only be obtained after the container is started
	at org.testcontainers.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:174)
	at org.testcontainers.containers.ContainerState.getMappedPort(ContainerState.java:141)
	at org.testcontainers.containers.MockServerContainer.getServerPort(MockServerContainer.java:50)	

With the mockServer.start() in the @BeforeEach I see for some of the tests:

[ForkJoinPool-1-worker-71] INFO docker[jamesdbloom/mockserver:mockserver-5.10.0] - Creating container for image: jamesdbloom/mockserver:mockserver-5.10.0

Container startup failed
org.testcontainers.containers.ContainerLaunchException: Container startup failed
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:330)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)
	at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.start(TestcontainersExtension.java:242)
...
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:323)
	... 59 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:497)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:325)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	... 60 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://localhost:49711/mockserver/status should return HTTP [200])
	at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:226)
	at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
	at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:892)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:440)
	... 62 more

I guess too many containers are started at the same time? Restarting docker "fixed" it.

An additional 'mockserver.maxSocketTimeout': '120000' in the systemProperties fixed the then (in some tests) occurring org.mockserver.client.SocketCommunicationException: Response was not received from MockServer after 20000 milliseconds, to wait longer please use "mockserver.maxSocketTimeout" system property or ConfigurationProperties.maxSocketTimeout(long milliseconds).

But from time to time, a test fails due to a connection error:, e.g. java.net.ConnectException: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:49305. How could we fix this?

In general, this results for a smaller project in a speedup from 12m 45s to 4m 57s (295 total tests) on our CI server and on my machine from ca. 20m to 2.5m!

[bsideup]

@mfbieber have you considered making the container static first? It looks like you were starting a container per test. Even without the parallel mode, just making the container static would already give a nice test time reduction.

Also, the errors you're getting seem to be related to container startup failures. They may happen essentially, as you're trying to start too many things that once, and they are getting OOM-ed, for example.

I would really recommend optimizing your setup (being able to start a container per test in an option, not a must) first.

[mfbieber]

@mfbieber have you considered making the container static first? It looks like you were starting a container per test. Even without the parallel mode, just making the container static would already give a nice test time reduction.

Yes, one container per test is okay (and expected from what I described above).

And yes, I tried the static and not concurrent approach and the speedup is not that drastic and all tests but the first test fail. I guess the container answers only for the first test with the desired answer.

I still don't understand how the mock server will know what to answer if I just have one container functioning as the mock server. I experienced similar issues with the suggested singleton AbstractBaseTestContainer approach.

(The execution of the tests is what slows down everything and is, in this case, a result of a retry mechanism on HTTP error codes I am testing - I expect each test to run for ca. 1 minute).

And @cowwoc also wanted this parallelism, as I understand his comment:

@bsideup My goal is to run independent tests in parallel and ensure as much as possible that one test cannot impact others.

Consequently, while I don't really care whether I use a single or separate JVMs the latter is preferable because it guarantees that tests cannot impact one another. Regardless of whether we have a single or multiple JVMs, I need each test to run against a separate TestContainer instance.

[bsideup]

I still don't understand how the mock server will know what to answer if I just have one container functioning as the mock server.

@mfbieber as the mock server process will be reused between the tests, you would obviously need to ensure that one test does not affect another. This can be done by either clearing MockServer's expectations (see https://mock-server.com/mock_server/clearing_and_resetting.html ) or by using per-test URL prefixes, so that each test will work with a unique set of URLs. This is outside of Testcontainers' responsibilities, as we only provide a container definition for MockServer.

Starting 10s of 100s of containers per test session is possible, but expensive (time and resources wise). MockServer is implemented in Java and (inside the container) runs a Java process that consumes a good amount of memory (400BM or so, last time I checked). Now, if you have 8 CPUs, you will be running 8 tests in parallel, each starting and stopping 400MB process, resulting in 3.2Gb of memory allocated. Some Docker installations may not have that much memory.

tl;dr: Testcontainers does not include any magic and an attempt to start more containers than you have resources on your machine will result in an error (from the containers' side, not in Testcontainers). There is more: starting 4 or 8 (just an example based on a common number of virtual CPU cores available) heavyweight containers at once may result in timeouts because it would consume a lot of CPU.

[mfbieber]

@bsideup thanks for your reply and explanations. I am aware that it is resource expensive to run that many containers at once.

Since running containers in parallel is possible (and we could close this issue), how about updating the documentation and providing configuration options for doing that?

The static or singleton approach really didn't result in a significant speedup. Having to provide per-test URL prefixes, or ensure otherwise that the mock responses don't affect each other, is also very laborious.

Regarding the connection timeout, could a .waitingFor(Wait.forHealthcheck()) help to wait for the container in the option with the 10s of containers running in parallel?