deploy jar silently clobbers jar resources having the same filename

[beasleyr-vmw]

trafficstars

Description of the problem / feature request:

When creating a deploy jar (bazel build foo_deploy.jar), if multiple constituent jars have resource files with identical names, only one appears in the deploy jar. This can catch users off guard, as their build succeeded but the bits will fail at runtime.

This is likely to happen when porting a project from Maven to Bazel, where the Maven deployment consists of individually deployed jars, and the developer is experimenting with Bazel's monolithic deploy jar.

Bazel ought to at least emit a warning when one file clobbers another while assembling a deploy jar, if it doesn't terminate the build altogether.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

#!/bin/bash

set -e

## Create project
# WORKSPACE and top-level java_binary
touch WORKSPACE
cat > BUILD <<EOF
java_binary(
    name = "resource_demo",
    srcs = glob(["src/**/*.java"]),
    runtime_deps = [
        "//subproj1",
        "//subproj2",
    ],
)
EOF

mkdir -p src/main/java
cat > src/main/java/resource_demo.java <<EOF
class MainClass { public static void main(String args[]) {} }
EOF

# Subprojects
for proj in subproj1 subproj2; do
   mkdir -p ${proj}/src/main/resources
   echo $proj > ${proj}/src/main/resources/id.dat
   cat > $proj/BUILD <<EOF
java_library(
    name = "$proj",
    resources = ["src/main/resources/id.dat"],
    visibility = ["//visibility:public"],
)
EOF
done

bazel_bin=$(bazel info bazel-bin)
bazel build :resource_demo_deploy.jar
echo "subproj1"
zipinfo -1 ${bazel_bin}/subproj1/libsubproj1.jar | sed 's/^/    /'
echo "subproj2"
zipinfo -1 ${bazel_bin}/subproj2/libsubproj2.jar | sed 's/^/    /'
echo "resource_demo_deploy.jar"
zipinfo -1 ${bazel_bin}/resource_demo_deploy.jar | sed 's/^/    /'
# subproj1's id.dat is present. No indication that subproj2 had a conflicting id.dat.
echo "Examining id.dat"
unzip -c ${bazel_bin}/resource_demo_deploy.jar id.dat

What operating system are you running Bazel on?

CentOS 6.6 and 7.2 macOS High Sierra

What's the output of bazel info release?

"development version"

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

We track Bazel release tags, apply a few localizations / pending PR patches, and build from source.

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

n/a

Have you found anything relevant by searching the web?

Sort of. This bazel-discuss thread touches on detecting duplicated .class files: https://groups.google.com/forum/#!topic/bazel-discuss/aDoI3XTSyWA

Alex Humesky guided the original author to Singlejar.java, indicating this would be the place to start if interested in modifying Bazel.

Any other information, logs, or outputs that you want to share?

Singlejar has a --warn_duplicate_resources option which Bazel uses when creating source jars. Perhaps this flag should be passed in from DeployArchiveBuilder.java?

[ghost]

It looks like passing --no_duplicates to singlejar will cause the build to terminate:

singlejar: src/tools/singlejar/output_jar.cc:413: id.dat is present both in .../bin/subproj1/libsubproj1.jar and .../bin/subproj2/libsubproj2.jar

So how about --

1. tweaking DeployArchiveBuilder.java to pass --no_duplicates by default; and
2. if users complain, add and plumb through a java_binary.deploy_ignore_duplicates attr to allow users to restore the old behavior on a per-target basis?

diff --git a/src/main/java/com/google/devtools/build/lib/rules/java/DeployArchiveBuilder.java b/src/main/java/com/google/devtools/build/lib/rules/java/DeployArchiveBuilder.java
index a07d55df03..8dc91a6f83 100644
--- a/src/main/java/com/google/devtools/build/lib/rules/java/DeployArchiveBuilder.java
+++ b/src/main/java/com/google/devtools/build/lib/rules/java/DeployArchiveBuilder.java
@@ -211,6 +211,7 @@ public class DeployArchiveBuilder {
       args.add("--compression");
     }
     args.add("--normalize");
+    args.add("--no-duplicates");
     if (javaMainClass != null) {
       args.add("--main_class", javaMainClass);
     }

Or is this something that should be hidden behind an --experimental flag for a release?

[exchgr]

Is there a way to construct a jar of jars instead of the way it's done now?

[ittaiz]

We do that in our own custom starlark rule.

[kamalmarhubi]

@ittaiz do you use onejar or something else?

[softprops]

How are others addressing this using bazel native deploy jar?

[softprops]

Note for others that bump into this found hack that works but take care in using it as its a hack and depends on undocumented features of bazel, namely that if you put your files under a file that falls under a small curated list that bazel supports everything "just works"

In my case if I drop my resource under META-INF/services/,a conventional directory used by java ServiceLoader you'll get merged resource out of the box with no additional configuration.

[softprops]

I just ran into another issue but one I'm unable to control. with log4j 2 which Im assuming is ubiquitously used with java no

log4j writes available plugins to a file called META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat which isn't on the list small curated list that bazel supports

so when I build an uber jar I'm not seeing the concatenated list of plugins

bazel build //foo:bar_deploy.jar                        
jar xvf  bazel-bin/foo/bar_deploy.jar  META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat
cat META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat # < does not contain all the plugins that were on the classpath prior to uber jarring

this leads to runtime errors like ERROR StatusLogger Unable to locate plugin for JsonTemplateLayout and so forth when debugging with -Dlog4j2.debug=true

how are others working around this with bazel?

[softprops]

This log4j change which moves files to javas service loader for may help because that’s one of bazels undocumented directories that will merge resources but this change won’t land in a release until log4j3 in its release tracker

https://issues.apache.org/jira/plugins/servlet/mobile#issue/LOG4J2-2621

[softprops]

Doing a bit more research on this merging this resource is a bit more than concatenating files

it turns out there are several solutions for this for various build tools but surprisingly I could find none for bazel. only mvn, gradle, sbt, and leiningen communities have solutions for this

https://github.com/search?q=Log4j2Plugins.dat&type=

Surely there must be some cross section between production jvm users using log4j2 and bazel to build deploy jars out there

[github-actions[bot]]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!

[jwilliams-ocient]

No help from @bazelbuild/triage on this one? Running into the exact same issue here with log4j...

[jmmv]

This was marked as "not stale". Should the bug be reopened? This is still a problem.

[fmeum]

@iancha1992 Could you reopen? I lack the permission to do so.