Coverage fails with missing *.jacoco_metadata.txt for Java 17

[gergelyfabian]

If I run coverage while using Java 17 it fails with an error:

Discovery starting.
Discovery completed in 87 milliseconds.
Run starting. Expected test count is: 1
TestSuite:
things
- should work (32 milliseconds)
Run completed in 200 milliseconds.
Total number of tests run: 1
Suites: completed 2, aborted 0
Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
java.io.FileNotFoundException: /home/user/.cache/bazel/_bazel_user/.../sandbox/linux-sandbox/10/execroot/scala_example/bazel-out/k8-fastbuild/bin/example-maven/test.runfiles/scala_example/example-maven/test.jacoco_metadata.txt (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
	at com.google.testing.coverage.jarjar.com.google.common.io.Files$FileByteSource.openStream(Files.java:130)
	at com.google.testing.coverage.jarjar.com.google.common.io.Files$FileByteSource.openStream(Files.java:120)
	at com.google.testing.coverage.jarjar.com.google.common.io.ByteSource$AsCharSource.openStream(ByteSource.java:460)
	at com.google.testing.coverage.jarjar.com.google.common.io.CharSource.readLines(CharSource.java:359)
	at com.google.testing.coverage.jarjar.com.google.common.io.Files.readLines(Files.java:558)
	at com.google.testing.coverage.JacocoCoverageRunner.getFilesFromFileList(JacocoCoverageRunner.java:355)
	at com.google.testing.coverage.JacocoCoverageRunner.access$100(JacocoCoverageRunner.java:81)
	at com.google.testing.coverage.JacocoCoverageRunner$2.run(JacocoCoverageRunner.java:546)

Reproduction is here: https://github.com/gergelyfabian/bazel-scala-example/tree/java_17

git clone https://github.com/gergelyfabian/bazel-scala-example
git checkout java_17
# This will produce the above error:
bazel coverage //...

bazel test //... works for all targets. This branch uses Bazel 5.1.1.

For a comparison with Bazel 5.1.1 and Java 11 coverage works:

git clone https://github.com/gergelyfabian/bazel-scala-example
git checkout master
bazel coverage //...

The only difference between the two branches is that java_17 has .bazelrc changes:

-build --java_language_version=11
-build --java_runtime_version=remotejdk_11
+build --java_language_version=17
+build --java_runtime_version=remotejdk_17

[gergelyfabian]

Interestingly, the file that is missing for Java 17 is also not there for Java 11:

$ ls bazel-out/k8-fastbuild/bin/example-maven/test.runfiles/scala_example/example-maven/
example-maven.jar  example-maven-offline.jar  test  test.args  test.jar  test_wrapper.sh

[gergelyfabian]

I've modified JacocoCoverageRunner locally to add some debugging:

                  System.out.println("metadataFileFinal: " + metadataFileFinal);
                  System.out.println("metadataFilesFinal: " + (metadataFilesFinal == null ? null : Arrays.toString(metadataFilesFinal)));

                  if (metadataFileFinal != null || metadataFilesFinal != null) {
                    File[] metadataJars;
                    if (metadataFilesFinal != null) {
                      metadataJars = metadataFilesFinal;
                    } else {
                      metadataJars =
                          hasOneFileFinal
                              ? new File[] {new File(metadataFileFinal)}
                              : getFilesFromFileList(new File(metadataFileFinal), javaRunfilesRoot)
                                  .toArray(new File[0]);
                    }

Built this and used as a custom jacocorunner.

This prints for Java 11:

metadataFileFinal: /home/user/opt/bazel-scala-example/tools/JacocoCoverage_jarjar_deploy.jar
metadataFilesFinal: [/home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/execroot/scala_example/bazel-out/k8-fastbuild/bin/example-lib/example-lib-offline.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/execroot/scala_example/bazel-out/k8-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/execroot/scala_example/bazel-out/k8-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/org/scala-lang/scala-reflect/2.13.8/scala-reflect-2.13.8.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalactic/scalactic_2.13-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalatest/scalatest_2.13-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalatest_compatible/scalatest-compatible-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalatest_core/scalatest-core_2.13-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalatest_featurespec/scalatest-featurespec_2.13-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalatest_flatspec/scalatest-flatspec_2.13-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalatest_freespec/scalatest-freespec_2.13-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalatest_funspec/scalatest-funspec_2.13-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalatest_funsuite/scalatest-funsuite_2.13-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalatest_matchers_core/scalatest-matchers-core_2.13-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalatest_mustmatchers/scalatest-mustmatchers_2.13-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scalatest_shouldmatchers/scalatest-shouldmatchers_2.13-3.2.9.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/execroot/scala_example/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_scala/scala/support/test_reporter.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/external/io_bazel_rules_scala_scala_xml/scala-xml_2.13-1.3.0.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/execroot/scala_example/bazel-out/host/bin/external/io_bazel_rules_scala/src/java/io/bazel/rulesscala/scala_test/librunner.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/execroot/scala_example/bazel-out/host/bin/external/bazel_tools/tools/java/runfiles/librunfiles.jar, /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/execroot/scala_example/bazel-out/k8-fastbuild/bin/example-lib/test.jar, /home/user/opt/bazel-scala-example/tools/JacocoCoverage_jarjar_deploy.jar]

For Java 17:

metadataFileFinal: /home/user/.cache/bazel/_bazel_user/64527517dacf7170f7d914889f83481c/sandbox/linux-sandbox/161/execroot/scala_example/bazel-out/k8-fastbuild/bin/example-maven/test.runfiles/scala_example/example-maven/test.jacoco_metadata.txt
metadataFilesFinal: null

So it seems for Java 17 metadataFilesFinal becomes null and hence it runs in to the method that looks for a file that is not there.

[gergelyfabian]

In JacocoCoverageRunner:

  private static URL[] getUrls(ClassLoader classLoader, boolean wasWrappedJar) {
    URL[] urls = getClassLoaderUrls(classLoader);
    System.out.println("wasWrappedJar: " + wasWrappedJar);
    System.out.println("urls: " + (urls == null ? null : Arrays.toString(urls)));

Prints for Java 11:

wasWrappedJar: false
urls: [...long list of jars...]

For Java 17:

wasWrappedJar: false
urls: null

[gergelyfabian]

Modified another method in JacocoCoverageRunner:

  private static URL[] getClassLoaderUrls(ClassLoader classLoader) {
    if (classLoader instanceof URLClassLoader) {
      System.out.println("classLoader is instanceof URLClassLoader");
      return ((URLClassLoader) classLoader).getURLs();
    }

    // java 9 and later
    if (classLoader.getClass().getName().startsWith("jdk.internal.loader.ClassLoaders$")) {
      System.out.println("classLoader.getClass().getName().startsWith(\"jdk.internal.loader.ClassLoaders$\")");
      try {
        Field field = Unsafe.class.getDeclaredField("theUnsafe");
        field.setAccessible(true);
        Unsafe unsafe = (Unsafe) field.get(null);

        // jdk.internal.loader.ClassLoaders.AppClassLoader.ucp
        Field ucpField = classLoader.getClass().getDeclaredField("ucp");
        long ucpFieldOffset = unsafe.objectFieldOffset(ucpField);
        Object ucpObject = unsafe.getObject(classLoader, ucpFieldOffset);

        // jdk.internal.loader.URLClassPath.path
        Field pathField = ucpField.getType().getDeclaredField("path");
        long pathFieldOffset = unsafe.objectFieldOffset(pathField);
        ArrayList<URL> path = (ArrayList<URL>) unsafe.getObject(ucpObject, pathFieldOffset);

        return path.toArray(new URL[path.size()]);
      } catch (Exception e) {
        System.out.println("Detected exception: " + e);
        e.printStackTrace();
        return null;
      }
    }
    System.out.println("Returning default null");
    return null;
  }

Then for Java 11:

classLoader.getClass().getName().startsWith("jdk.internal.loader.ClassLoaders$")
wasWrappedJar: false
urls: [...long list...]

For Java 17:

classLoader.getClass().getName().startsWith("jdk.internal.loader.ClassLoaders$")
Detected exception: java.lang.NoSuchFieldException: ucp
java.lang.NoSuchFieldException: ucp
	at java.base/java.lang.Class.getDeclaredField(Class.java:2610)
	at com.google.testing.coverage.JacocoCoverageRunner.getClassLoaderUrls(JacocoCoverageRunner.java:413)
	at com.google.testing.coverage.JacocoCoverageRunner.getUrls(JacocoCoverageRunner.java:371)
	at com.google.testing.coverage.JacocoCoverageRunner.main(JacocoCoverageRunner.java:443)
wasWrappedJar: false
urls: null

So it seems that loading the classloader urls fails with an Exception due to the "ucp" field missing.

Is this a bug with rules_scala or bazel itself? (I saw that for Java targets' coverage we are not running on this code parts at all)

[gergelyfabian]

JacocoCoverageRunner debugging changes are here:

https://github.com/gergelyfabian/bazel/tree/jacocorunner_debugging

Built JacocoCoverageRunner with:

bazel build src/java_tools/junitrunner/java/com/google/testing/coverage:JacocoCoverage_jarjar_deploy.jar
cp bazel-bin/src/java_tools/junitrunner/java/com/google/testing/coverage/JacocoCoverage_jarjar_deploy.jar ~/opt/bazel-scala-example/tools/JacocoCoverage_jarjar_deploy.jar && chmod +w ~/opt/bazel-scala-example/tools/JacocoCoverage_jarjar_deploy.jar

In bazel-scala-example select java_17 branch and uncomment jacocorunner option in Java and Scala toolchains.

[gergelyfabian]

@liucijus , what do you think, is this a rules_scala or Bazel bug?

[gergelyfabian]

Steps to reproduce with custom jacocorunner changes:

cd ~/opt
git clone https://github.com/gergelyfabian/bazel-scala-example
git clone https://github.com/gergelyfabian/bazel
cd bazel
git checkout jacocorunner_debugging
bazel build src/java_tools/junitrunner/java/com/google/testing/coverage:JacocoCoverage_jarjar_deploy.jar
cp bazel-bin/src/java_tools/junitrunner/java/com/google/testing/coverage/JacocoCoverage_jarjar_deploy.jar ~/opt/bazel-scala-example/tools/JacocoCoverage_jarjar_deploy.jar && chmod +w ~/opt/bazel-scala-example/tools/JacocoCoverage_jarjar_deploy.jar
cd ~/opt/bazel-scala-example
git checkout java_17
# Uncomment jacocorunner parameters for Scala and Java toolchains in tools/jdk/BUILD.bazel.
vi tools/jdk/BUILD.bazel
# Run coverage:
bazel coverage //...

[gergelyfabian]

Can the following issue and fix be related?

https://github.com/Col-E/Recaf/issues/312 https://github.com/Col-E/Recaf/commit/2561d92d72227671655597d6ad0f137ae2802a1f

[gergelyfabian]

This is a Bazel bug: https://github.com/bazelbuild/bazel/pull/15081, and was already fixed on master.

I guess a workaround is to rebuild jacocorunner from the fixed Bazel version and use that as a custom jacocorunner.

[gergelyfabian]

Use the updated scripts/build_jacocorunner/build_jacocorunner_bazel_5.0+.sh script to build a custom jacocorunner to work this issue around (will be merged in #1399, hopefully).