JUnit5: Start containers concurrently

[OLibutzki]

When using Testcontainers in combination with JUnit5 using @Testcontainers and @Container all the containers are started sequentially.

Especially when there are multiple containers involved it could save some time to start them in parallel.

This feature request is about facilitating the concurrent start of containers.

[kiview]

Hi @OLibutzki, thanks for the suggestion, this is correct.

Note that you can work around this issue, if you use Testcontainers directly and without the JUnit5 extension, e.g. like:

@BeforeAll
static void setup() {
  Stream.of(container1, container2).parallel().forEach(GenericContainer::start);
}

In most cases, the direct use of Testcontainers without test framework integration is in no way inferior.

[bountin]

@kiview Nice idea!

I just have struggled to get it working with DockerComposeContainer. With an instance of that one, or to be precise, mixing GenericContainer and it, blocks forever leaving me without a clue in which state the worker threads are. Here's a minimal repo for reproduction: https://github.com/bountin/testcontainer-parallel-startup (./gradlew test)

Same behavior with a manual Threadpool (newFixedThreadPool, submit, submit, awaitTermination). It times out the await, even though I can step through both start methods completely in the debugger. Maybe there is some resource that keeps the thread from finally returning?

[kiview]

Thanks for the reproducer @bountin.

I can confirm that your example hangs. After adding logback and configuring logging, I could see that Testcontainers seems to hang after starting the socat container (which is used internally by DockerComposeContainer). Since we generally use this pattern without problems, this seems to be explicitly a problem with regards to the implementation of DockerComposeContainer (or the internal SocatContainer). It is also very curious to see, that none of our timeouts catch this.

For now, I would simply recommend against the use of DockerComposeContainer in this way, until we figured out the root cause.

Note that you should also add the exposed ports, so Testcontainers can use a meaningful default wait strategy (but this is unrelated to seeing the hanging behavior).

Edit: It also seems to work fine when the startup is moved out of the static block, e.g. into a test method. So what I can observe is something hanging when we start multiple containers in paralell, with one being DockerComposeContainer, in a static block of a JUnit Jupiter test class.

[40rn05lyv]

It's possible to use both @Container annotation with a parallel approach. You can implement org.testcontainers.lifecycle.Startable interface that would have several containers inside it. Like this:

public class ParallelContainers implements org.testcontainers.lifecycle.Startable {

    private final GenericContainer container1 = ...;
    private final GenericContainer container2 = ...;

    @Override
    public void start() {
        Stream.of(this.container1, this.container2).parallel().forEach(GenericContainer::start);
    }

    @Override
    public void stop() {
        Stream.of(this.container1, this.container2).parallel().forEach(GenericContainer::stop);
    }

}

And then just use @Container ParallelContainers containers in test classes.

[cjmartinez217]

Hey I am a first time contributor and would love to take a crack at this!

[cjmartinez217]

If there is any insight on how to approach developing this feature rather than just a work around that would be much appreciated

[Olex1313]

@cjmartinez217, If you still wanting to contribute can I somehow help you? Otherwise I can try to work on the implementation idea provided by @40rn05lyv

[kiview]

Sorry @cjmartinez217 and @Olex1313 for not responding earlier. This is essentially up for grabs but would require some exploration of JUnit5 extension best practices and a possible API.

I imagine it could be a flag on the @Testcontainers extension annotation. The implementation code could like make use of the Stream.of(this.container1, this.container2).parallel().forEach(GenericContainer::stop); example. Not necessary to implement a new Startable class.

[Olex1313]

@kiview Well sorry for long thinking about the problem, maybe it instead could be a flag on @Container annotation, so in the https://github.com/testcontainers/testcontainers-java/blob/main/modules/junit-jupiter/src/main/java/org/testcontainers/junit/jupiter/TestcontainersExtension.java, we might split containers into 2 groups every time we have to start it, and as you said start ones with the flag with sample of code like this

parallelContainers.parallel().forEach(GenericContainer::start);

?

[samed-bicer]

@kiview I have tried to come up with a solution and would like to get some review, I am more than happy to contribute if you can help with the review and the direction I need to take.

[Kamveno]

Hi I am new contributor here and am an enthusiast to work with you to crack