JUnit 5 support

[BlackIsTheNewBlack]

ATTENTION! Please read and follow:

• if this is a question about how to build / test / query / deploy using Bazel, ask it on StackOverflow instead: https://stackoverflow.com/questions/tagged/bazel
• if this is a discussion starter, send it to [email protected]
• if this is a bug or feature request, fill the form below as best as you can.

Description of the problem / feature request:

JUnit v5 (Jupiter) is a significant new version of JUnit, bringing many new features and changes.

Bazel should natively support JUnit v5 (Jupiter) tests. JUnit v5 annotations and assertions cannot be run in v4.

Feature requests: what underlying problem are you trying to solve with this feature?

"bazel test" should run for JUnit5 tests

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Replace this line with your answer.

What operating system are you running Bazel on?

Windows.

What's the output of bazel info release?

0.19.1

Have you found anything relevant by searching the web?

https://github.com/JeffreyFalgout/bazel-junit5.git It fails with all kind of errors.

Any other information, logs, or outputs that you want to share?

Will upload sample on request.

[irengrig]

P1 because I consider this important feature.

[lberki]

No can do, at least not with the urgency that P1 bugs need although I do agree that we should do this eventually.

[jbduncan]

Good news: The JUnit 5 team have a sample for using JUnit 5 in Bazel! :)

https://github.com/junit-team/junit5-samples/tree/master/junit5-jupiter-starter-bazel

[jbduncan]

Sure, it's not native, but it at least (should) fix the problem that the other existing example, https://github.com/JeffreyFalgout/bazel-junit5.git, doesn't work.

[BlackIsTheNewBlack]

Thank you @irengrig and @lberki Thanks @jbduncan for the tip

[raz-canva]

Any update on a native solution for this?

[martindow]

The JUnit team's Bazel starter project (https://github.com/junit-team/junit5-samples/tree/master/junit5-jupiter-starter-bazel) only appears to work with the Java 8 Bazel toolchain.

This works fine (as does not specifying java_toolchain): bazel test --java_toolchain=@bazel_tools//tools/jdk:toolchain_java8 //...

Specifying any other toolchain causes the ConsoleLauncher to run, but the test log shows that it doesn't find any tests (so it reports the tests as passing, even when there are failures): bazel test --java_toolchain=@bazel_tools//tools/jdk:toolchain_java11 //... (It's the same problem with toolchain_java9 and toolchain_java10).

The log output shows that it ran, but found no tests:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test
-----------------------------------------------------------------------------
Thanks for using JUnit! Support its development at https://junit.org/sponsoring
╷
└─ JUnit Jupiter ✔
Test run finished after 27 ms
[         1 containers found      ]
[         0 containers skipped    ]
[         1 containers started    ]
[         0 containers aborted    ]
[         1 containers successful ]
[         0 containers failed     ]
[         0 tests found           ]
[         0 tests skipped         ]
[         0 tests started         ]
[         0 tests aborted         ]
[         0 tests successful      ]
[         0 tests failed          ]

Any clues as to what could be causing this? Maybe org.junit.platform.console.ConsoleLauncher's classpath doesn't have the test classes on it (they're declared in deps)?

[martindow]

Update on my issue above: This appears to be a problem with the homebrew installation of bazel, version 0.29.0-homebrew. I've since used bazelisk to install version 0.29.0 directly and it's running the tests as expected. I have no idea what the difference is between the two...

[martindow]

Might be useful to clarify for any other Bazel newbies like myself what the java_junit5_test rule in junit5.bzl (https://github.com/junit-team/junit5-samples/tree/master/junit5-jupiter-starter-bazel) is doing under the hood. It:

1. Declares the required JUnit5 Maven artifacts in WORKSPACE and
2. Creates a java_test rule with the main class set to org.junit.platform.console.ConsoleLauncher.

If you're using the new maven_install rule (https://github.com/bazelbuild/rules_jvm_external) and want to inline these steps into your project, then the following changes will cause the sample project to execute:

Add the maven_install rule to WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
RULES_JVM_EXTERNAL_TAG = "2.10"
RULES_JVM_EXTERNAL_SHA = "1bbf2e48d07686707dd85357e9a94da775e1dbd7c464272b3664283c9c716d26"
http_archive(
    name = "rules_jvm_external",
    strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG,
    sha256 = RULES_JVM_EXTERNAL_SHA,
    url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG,
)
load("@rules_jvm_external//:defs.bzl", "maven_install")
maven_install(
    artifacts = [
        "org.junit.jupiter:junit-jupiter-api:5.5.0",
        "org.junit.jupiter:junit-jupiter-engine:5.5.0",
        "org.junit.jupiter:junit-jupiter-params:5.5.0",
        "org.apiguardian:apiguardian-api:1.0.0",
        "org.opentest4j:opentest4j:1.1.1",
        "org.junit.platform:junit-platform-commons:1.5.0",
        "org.junit.platform:junit-platform-console:1.5.0",
        "org.junit.platform:junit-platform-engine:1.5.0",
        "org.junit.platform:junit-platform-launcher:1.5.0",
        "org.junit.platform:junit-platform-suite-api:1.5.0",
    ],
    repositories = [
        "http://central.maven.org/maven2",
    ],
)

Then change the test BUILD file to directly declare the java_test rule like this:

package(
    default_visibility = ["//src/test:__subpackages__"],
)

java_test(
    name = "junit5-jupiter-starter-bazel-test",
    srcs = glob([
        "*.java",
    ]),
    use_testrunner = False,
    main_class = "org.junit.platform.console.ConsoleLauncher",
    args = ["--select-package", "com.example.project"],
    deps = [
        "//src/main/java/com/example/project:junit5-jupiter-starter-bazel",
        "@maven//:org_junit_jupiter_junit_jupiter_api",
        "@maven//:org_junit_jupiter_junit_jupiter_engine",
        "@maven//:org_junit_jupiter_junit_jupiter_params",
        "@maven//:org_junit_platform_junit_platform_suite_api",
        "@maven//:org_apiguardian_apiguardian_api",
        "@maven//:org_opentest4j_opentest4j",
    ],
    runtime_deps = [
        "@maven//:org_junit_platform_junit_platform_commons",
        "@maven//:org_junit_platform_junit_platform_console",
        "@maven//:org_junit_platform_junit_platform_engine",
        "@maven//:org_junit_platform_junit_platform_launcher",
        "@maven//:org_junit_platform_junit_platform_suite_api",
    ],
)

Might be useful to someone...

[guw]

@martindow Any luck experimenting with --reports-dir argument for getting a nice report of tests for Jenkins (and other CI systems)?

[PeterIzso]

@lberki @iirina Are there any news on the native support?

[unoexperto]

For those who curious following example worked for me in bazel 2.x https://github.com/bmuschko/bazel-examples/tree/master/java/junit5-test

[guw]

@unoexperto I also found a solution in case you want to expose/collect the JUnit reports for Jenkins.

https://github.com/salesforce/bazel-maven-proxy/tree/master/tools/junit5

[razfriman]

Bump - Would be great to have first class support for this

[asinbow]

We have built a workaround for it: https://github.com/junit-team/junit5-samples/pull/133 Please take a look. A blog post: https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068

[smicha]

I know it is not exactly a bazel issue. Do we know if the junit folks are working on bringing junit5 support natively to bazel anytime soon .

[MikhailTymchukFT]

Any timeline for this feature? We know it's P3, but JUnit 4 becomes rather old.

[fivetran-noopurbathija]

What is the timeline for this issue? Its been open since 2018.

[razfriman]

Bump

[chrismgrayftsinc]

So I finally figured out why the behavior @martindow noticed above is happening (that is, no tests being run for toolchains other than java8). The reason is the version of java in the runfiles directory is lower than the version of java that compiled the classes. Running the test classes directly through ConsoleLauncher gives an error like Caused by: java.lang.UnsupportedClassVersionError: com/foo/bar/FooTest has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0.

This problem seems to exist in bazel versions < 4, but unfortunately the latest bazel docker image is 3.5.0 so it's still relevant to us, since we use the docker image in our CI.

The next question is whether it is possible to force the version of java in the runfiles directory to a higher version.

[chrismgrayftsinc]

For users of the 3.5.0 docker version, the following bazelrc works for me:

test --define=ABSOLUTE_JAVABASE=/usr/lib/jvm/11.29.3-ca-jdk11.0.2/reduced/
test --javabase=@bazel_tools//tools/jdk:absolute_javabase
test --host_javabase=@bazel_tools//tools/jdk:absolute_javabase
test --java_runtime_version=@bazel_tools//tools/jdk:absolute_javabase
test --java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla
test --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla

I put this in a file named .bazel-ci.rc and invoke the tests with bazel --bazelrc=.bazel-ci.rc test --test_output=errors //:foo-tests

[shs96c]

I have a PR open for rules_jvm_external that adds a junit5 runner and java_junit5_test: https://github.com/bazelbuild/rules_jvm_external/pull/629 This is based on what we've been using extensively at work, and has a number of advantages:

• java_junit5_test is a drop-in replacement for java_test
• Integrates with most of the features of the bazel test runner (eg. test filtering)
• Works fine in the IJ plugin: the tests are just picked up as regular bazel tests, and stdout and stderr are captured properly
• We handle System.exit being called and prevent that taking out the test runner

The major difference between this and the other approaches is that we don't depend on the ConsoleLauncher, so we have more control over how outputs are generated and placed, and how tests are run.

[TRL-Official]

I have a PR open for rules_jvm_external that adds a junit5 runner and java_junit5_test: bazelbuild/rules_jvm_external#629 This is based on what we've been using extensively at work, and has a number of advantages:

• java_junit5_test is a drop-in replacement for java_test
• Integrates with most of the features of the bazel test runner (eg. test filtering)
• Works fine in the IJ plugin: the tests are just picked up as regular bazel tests, and stdout and stderr are captured properly
• We handle System.exit being called and prevent that taking out the test runner

The major difference between this and the other approaches is that we don't depend on the ConsoleLauncher, so we have more control over how outputs are generated and placed, and how tests are run.

Hi @shs96c Thanks for this work. But I have one question. Your implementation of this JUnit5 Runner supports nested test classes? For example:

class SomeTest {
  @Nested
  @DisplayName("")
  class SomeFirstLevelNestedTest {
    @Nested
    @DisplayName("")
    class SomeSecondLevelNestedTest {
      @Nested
      @DisplayName("")
      class SomeThirdLevelNestedTest {
        // and so on
        // nesting of internal tests can be quite deep
        @Test
        void name() {
          Assertions.assertTrue(true);
        }
      }
    }
  }
}  

[TRL-Official]

We have built a workaround for it: junit-team/junit5-samples#133 Please take a look. A blog post: https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068

Hi @asinbow
Thanks for this work. But I have one question. Your implementation of this JUnit5 Runner supports nested test classes? For example:

class SomeTest {
  @Nested
  @DisplayName("")
  class SomeFirstLevelNestedTest {
    @Nested
    @DisplayName("")
    class SomeSecondLevelNestedTest {
      @Nested
      @DisplayName("")
      class SomeThirdLevelNestedTest {
        // and so on
        // nesting of internal tests can be quite deep
        @Test
        void name() {
          Assertions.assertTrue(true);
        }
      }
    }
  }
}  

[amilamanoj]

Hi @comius, Do you have any updates on this?

[shs96c]

There is a JUnit5 runner that ships as part of https://github.com/bazel-contrib/rules_jvm

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[mkobit]

Looking at some of the issues referencing this (like internal Google dependencies), it appears that this should eventually be added a native feature. Should this be unmarked as stale, then?

[shs96c]

I'd be more than happy if the junit5 runner from contrib_rules_jvm was moved into Bazel, but my feeling from the presentations that Google has been doing is that more and more things are being pushed out of Bazel itself and into rulesets. If that's the case, leaving this issue closed seems reasonable, since there's already an existing solution to the problem.

[daniel-b2c2]

Can somebody clarify please:

• what is the official route for running JUnit5 with bazel? The thread seems to point to this
• If so, does the above include support for nested tests?