[Bug]: No tests found with JUnit 5.9.0

[ilgrosso]

Module

Core

Testcontainers version

1.17.3

Using the latest Testcontainers version?

Yes

Host OS

Any OS

Host Arch

Any arch

Docker version

Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:02:57 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:03 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.7
  GitCommit:        0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb
 runc:
  Version:          1.1.3
  GitCommit:        v1.1.3-0-g6724737
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

Any tests annotated with @TestContainers not found by Maven Surefire plugin 2.22.0 with JUnit 5.9.0. All works fine with JUnit 5.8.2.

Relevant log output

No response

Additional Information

No response

[kiview]

Please share a reproducer. You still need to annotate the tests with the actual JUnit5 annotations. Also, it is @Testcontainers.

[ilgrosso]

yup @kiview sorry for typo about @Testcontainers

What I was meaning is that I do have some tests annotated by @Testcontainers which are running fine when

      <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter</artifactId>
        <version>5.8.2</version>
        <scope>test</scope>
      </dependency>

If I simply upgrade to 5.9.0 then such tests are not detected.

I'll try to come up shortly with a reproducer.

[kiview]

Alright, I think this is actually an issue with Surefire and JUnit5 and has nothing to do with Testcontainers. At least this is my assumption. Which version of Surefire are you using? Try to use the latest version, potentially a milestone release, such as 3.0.0-M7.

[ilgrosso]

Here you go: https://github.com/ilgrosso/testcontainers5680

As you can read from pom.xml, the default versions are

    <junit.version>5.8.2</junit.version>
    <surefire.version>2.22.2</surefire.version>

So, when you run

mvn clean test

you obtain

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running net.tirasa.test.testcontainers5680.ReproducerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.809 s - in net.tirasa.test.testcontainers5680.ReproducerTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

but when you go with

mvn clean test -Djunit.version=5.9.0

then you get

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

with

mvn clean test -Djunit.version=5.9.0 -Dsurefire.version=3.0.0-M7

you receive

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M7:test (default-test) on project testcontainers5680: 
[ERROR] 
[ERROR] Please refer to /home/ilgrosso/work/testcontainers5680/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] There was an error in the forked process
[ERROR] TestEngine with ID 'junit-jupiter' failed to discover tests
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
[ERROR] TestEngine with ID 'junit-jupiter' failed to discover tests
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:701)
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:311)
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:268)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1334)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1167)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:931)
[ERROR]         at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:301)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:211)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:165)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:157)
[ERROR]         at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:121)
[ERROR]         at org.mvndaemon.mvnd.builder.SmartBuilderImpl.buildProject(SmartBuilderImpl.java:178)
[ERROR]         at org.mvndaemon.mvnd.builder.SmartBuilderImpl$ProjectBuildTask.run(SmartBuilderImpl.java:198)
[ERROR]         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[ERROR]         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[ERROR]         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[ERROR]         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[ERROR]         at java.base/java.lang.Thread.run(Thread.java:829)

which is basically the same.

[ilgrosso]

My wild guess is that there is some dependency between Testcontainers 1.17.3 and JUnit 5.8.2 as shown by

mvn dependency:tree -Djunit.version=5.9.0

[INFO] net.tirasa.test:testcontainers5680:jar:1.0-SNAPSHOT
[INFO] +- org.testcontainers:postgresql:jar:1.17.3:test
[INFO] |  \- org.testcontainers:jdbc:jar:1.17.3:test
[INFO] |     \- org.testcontainers:database-commons:jar:1.17.3:test
[INFO] +- org.testcontainers:junit-jupiter:jar:1.17.3:test
[INFO] |  +- org.testcontainers:testcontainers:jar:1.17.3:test
[INFO] |  |  +- junit:junit:jar:4.13.2:test
[INFO] |  |  |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] |  |  +- org.slf4j:slf4j-api:jar:1.7.36:test
[INFO] |  |  +- org.apache.commons:commons-compress:jar:1.21:test
[INFO] |  |  +- org.rnorth.duct-tape:duct-tape:jar:1.0.8:test
[INFO] |  |  |  \- org.jetbrains:annotations:jar:17.0.0:test
[INFO] |  |  +- com.github.docker-java:docker-java-api:jar:3.2.13:test
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.10.3:test
[INFO] |  |  \- com.github.docker-java:docker-java-transport-zerodep:jar:3.2.13:test
[INFO] |  |     +- com.github.docker-java:docker-java-transport:jar:3.2.13:test
[INFO] |  |     \- net.java.dev.jna:jna:jar:5.8.0:test
[INFO] |  \- org.junit.jupiter:junit-jupiter-api:jar:5.8.2:test
[INFO] |     +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] |     +- org.junit.platform:junit-platform-commons:jar:1.8.2:test
[INFO] |     \- org.apiguardian:apiguardian-api:jar:1.1.2:test
[INFO] \- org.junit.jupiter:junit-jupiter:jar:5.9.0:test
[INFO]    +- org.junit.jupiter:junit-jupiter-params:jar:5.9.0:test
[INFO]    \- org.junit.jupiter:junit-jupiter-engine:jar:5.9.0:test
[INFO]       \- org.junit.platform:junit-platform-engine:jar:1.9.0:test

as you can see there are both JUnit 5.9.0 (as configured) and 5.8.2 as transitive from org.testcontainers:testcontainers:jar:1.17.3

[ilgrosso]

Additional finding: if adding the following exclude:

    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>junit-jupiter</artifactId>
      <version>${testcontainers.version}</version>
      <exclusions>
        <exclusion>
          <groupId>org.junit.jupiter</groupId>
          <artifactId>junit-jupiter-api</artifactId>
        </exclusion>
      </exclusions>
      <scope>test</scope>
    </dependency>

then the test is run even with -Djunit.version=5.9.0

[kiview]

Hey @ilgrosso, thanks for your detailed triaging. You are completely right, this is an issue with org.testcontainers.junit-jupiter bringing in the transitive org.junit.jupiter:junit-jupiter-api:jar:5.8.2 dependency.

So either there might be a better way to build our junit-jupiter extension here (we would need to check on best practices around extensions in the community), or we just update the dependency itself (although this does not seem like the clean solution to me).

[nilsmahlstaedt]

IntelliJ IDEA 2022.2.2 (Community Edition) has no problems to find the check test when tasked to run all test in the ReproducerTest Testclass for JUnit-Jupiter 5.8.2 and 5.9.0

Class name > Test Marker left of Text Pane > Run 'ReproducerTest'

Tried a version in gradle (and without surefire), taking your repoducer test case. build.gradle

dependencies {
    testImplementation 'org.testcontainers:postgresql:1.17.3'
    testImplementation 'org.testcontainers:junit-jupiter:1.17.3'
    //testImplementation 'org.junit.jupiter:junit-jupiter:5.9.0'
    testImplementation 'org.junit.jupiter:junit-jupiter:5.8.2'
}

Running gradlew clean test resulted in one picked up test regardless of the used JUnit Version.

[edysli]

You are completely right, this is an issue with org.testcontainers.junit-jupiter bringing in the transitive org.junit.jupiter:junit-jupiter-api:jar:5.8.2 dependency.

Would switching the dependency declaration api 'org.junit.jupiter:junit-jupiter-api:5.9.1' to implementation help? Gradle's Java Library plugin explains the subtle difference between api and implementation.

[eddumelendez]

@edysli yes. However, the concern is doing that change could break existing projects due to the absence of junit-jupiter in the classpath.

[edysli]

@eddumelendez it will indeed break projects relying on this transitive dependency. However, they shouldn't do this and instead declare a dependency on org.junit.jupiter:junit-jupiter -- just like @ilgrosso did -- since they are using it directly. For me, this is OK to break, but I'll leave this up to the maintainers, of course. :blush:

[kiview]

junit-pioneer, which I consider a reference implementation of JUnit5 extensions, is using implementation as well: https://github.com/junit-pioneer/junit-pioneer/blob/main/build.gradle.kts#L59

[edysli]

Thanks @kiview for the nice example. Taking inspiration from it, I reworked how JUnit artifacts are included, in order to avoid using the api configuration, see #5985.

[alexanderankin]

This is an issue I ran into at work and exposing the conflicting dependency as api(platform( was the solution i settled on.

A couple of things I found was that actually, in some cases (e.g. not through multiple transitive deps' api references) you can use "compileOnly platform" and it will completely hide your junit version from the consumer. This seems to violate some api contract because it fails unpredictably and causes frustration. When it does, you can revert it, and ask the consumer to use enforcedPlatform instead of platform. both of these are not the gradle way of doing things. ergo, the solution being presented is what i went with.

The dependency in question was spring-boot-dependencies (and friends) - using the solution here, the consumer still prints the right spring banner, so gradle seems to be smart about using the consumers implementation(platform( before the libraries' api(platform('s.