Producing .srcjar with `java_proto_library`

[amit-traiana]

Hi everyone,

As I'm new to Bazel, I'm not complete sure that's a bug or more of a feature request. I will out-line the problem, and maybe you guys can help me out.

Description of the problem / feature request:

I'm using java_proto_library to compile protobuf files. The output is a JAR file with the compiled generated code. However, because the JAR is to be used in a different build system that using Maven, I also need that JAR to have a POM file. For that to happen I use the pom_file rule specified here by bazel-common. pom_file generate a POM file, and need a java_library as target. What I tried to do, is to wrap java_proto_library inside java_library like so:

proto_library(
    name = "service_a_proto",
    srcs = [
        "service.proto",
    ],
    visibility = ["//visibility:public"],
)

java_proto_library(
    name = "service_a_java_proto",
    deps = [":service_a_proto"],
)

java_library(
    name = "service_a_java",
    srcs = [":service_a_java_proto"],
    visibility = ["//visibility:public"],
)

pom_file(
    name = "pom",
    template_file = "//tools:pom-template.xml",
    targets = [":service_a_java"],
)

This however won't work, because '//service_a:service_a_java_proto' does not produce any java_library srcs files (expected .java, .srcjar or .properties). Is there any reason why java_proto_library won't produce a .srcjar?

I checked for how people implement pom_file in the wild, and it seems all of them using an external implementation and not the native java_proto_library.

Is there a way to output a srcjar from the native java_proto_library rule? and if not - is that something you will consider adding?

Thanks!

[davido]

This fragment (borrowed from Gerrit Code Review), proto/BUILD:

load("@rules_java//java:defs.bzl", "java_proto_library")
load("@rules_proto//proto:defs.bzl", "proto_library")

proto_library(
    name = "cache_proto",
    srcs = ["cache.proto"],
)

java_proto_library(
    name = "cache_java_proto",
    visibility = ["//visibility:public"],
    deps = [":cache_proto"],
)

would produce:

  $ bazel build proto:cache_java_proto
WARNING: Waiting for server process to terminate (waited 5 seconds, waiting at most 60)
Starting local Bazel server and connecting to it...
INFO: Writing tracer profile to '/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/command.profile.gz'
INFO: Invocation ID: 51f82acf-e019-48b9-ba17-d41d887010c0
INFO: Analyzed target //proto:cache_java_proto (24 packages loaded, 1155 targets configured).
INFO: Found 1 target...
INFO: Deleting stale sandbox base /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox
Target //proto:cache_java_proto up-to-date:
  bazel-bin/proto/libcache_proto-speed.jar
  bazel-bin/proto/cache_proto-speed-src.jar
INFO: Elapsed time: 13.987s, Critical Path: 0.85s
INFO: 1 process: 1 remote cache hit.
INFO: Build completed successfully, 2 total actions

As the name can suggest: bazel-bin/proto/cache_proto-speed-src.jar is the source jar you are looking for:

  $ unzip -t bazel-bin/proto/cache_proto-speed-src.jar
Archive:  bazel-bin/proto/cache_proto-speed-src.jar
    testing: META-INF/MANIFEST.MF     OK
    testing: com/google/gerrit/server/cache/proto/Cache.java   OK

[amit-traiana]

Hello @davido,

Thanks for the answer. I modified the original response after doing some more research. Even when using the rules from @rules_java instead of the native java_proto_library. I'm getting the same error. Your right, a src jar is indeed being created:

Target //service_a:service_a_java_proto up-to-date:
  bazel-bin/external/com_google_protobuf/libdescriptor_proto-speed.jar
  bazel-bin/external/com_github_scalapb_scalapb/libscalapb_proto-speed.jar
  bazel-bin/service_a/libservice_a_proto-speed.jar
  bazel-bin/service_a/service_a_proto-speed-src.jar
INFO: Elapsed time: 83.166s, Critical Path: 22.86s

But when I try to wrap this rule inside java_library, I still get the following:

java_library(
    name = "service_a_library",
    srcs = [":service_a_java_proto"],
    visibility = ["//visibility:public"],
)

`//service_a:service_a_java_proto' does not produce any java_library srcs files (expected .java, .srcjar or .properties)`

I'm assuming java_proto_library create a srcjar but the rules doesn't return it as object is so I can't wrap it inside another java rule?

Thanks!

[Yannic]

java_proto_library (and java_library) do not "return" the srcjar by default, i.e. srcs = [":service_a_java_proto"], will try to use the compiled jar as input (me thinks).

I haven't used pom_file, but from looking at its implementation, I think you should be able to add the java_proto_library to the list of targets.

proto_library(
    name = "service_a_proto",
    srcs = [
        "service.proto",
    ],
    visibility = ["//visibility:public"],
)

java_proto_library(
    name = "service_a_java_proto",
    deps = [":service_a_proto"],
    visibility = ["//visibility:public"],
)

pom_file(
    name = "pom",
    template_file = "//tools:pom-template.xml",
    targets = [":service_a_java_proto"],
)

[amit-traiana]

Hello @Yannic , thanks for the response :-)

The above code will indeed work, but will not generate a valid POM file. Let me share more details about why I want to use a POM file.

The JAR generated by java_proto_library contains only generated compiled code in it. Consuming is on any Java application and trying to use it will yield compilation error, because this code need the protobuf-java library in order to compile. So in order for the JAR to be a real Maven consumable JAR, you need a POM file that state protobuf-java as dependency.

If pom_file will target service_a_java_proto directly - an empty POM file will get created. With no dependencies. In order to create a POM file with reference to protobuf-java, you have to wrap it inside a java_library like so:

java_library(
    name = "service_a_java",
    srcs = [":service_a_java_proto"],
    deps = [
        "//dependencies/maven/artifacts/com/google/protobuf:protobuf-java",
    ],
    visibility = ["//visibility:public"],
)

(the //dependencies target is being auto generated by bazel-deps).

Now if I'll reference the above target into a pom_file, it will include the dependency for protobuf-java.

I don't use java_library often enough (just for Proto) but based on examples I saw online you can chain one java_library into another java_library? shouldn't I be able to do the same with java_proto_library because technically they both output JAR at the end?

The dagger team had the same issue. They solved it by using a different implementation for java_proto_library. This rule returns a srcjar file.

You mentioned early on that "by default" it won't return srcjar. Is it possible to force the rule to return srcjar somehow? I couldn't find a parameter that controls that option.

Thanks!

[Yannic]

Just to get this right: if you do

java_library(
    name = "service_a_java",
    deps = [
        "//dependencies/maven/artifacts/com/google/protobuf:protobuf-java",
    ],
    visibility = ["//visibility:public"],
)

you get a working pom that declares a dependency on protobuf-java (but there's no code to compile).

I see three options: pom_file takes a list of targets, so you could add ":service_a_java_proto", or you add the java_proto_library to the deps of java_library (Note: it only works in deps, not in srcs`).

The third option is the most correct one, but also a little more complicated. If you bazel build //path/to:service_a_java_proto, Bazel will happily compile the generated Java code, so it has a dependency on the protobuf runtime somewhere. However, that is coming from an implicit dependency and not the one from //dependencies/.... By default, this implicit dependency is coming from @com_google_protobuf (you have that somewhere in your WORKSPACE, or another rule is loading it for you), but you can override it to use the runtime from maven. Create a new BUILD file next to your WORKSPACE and add the following content:

proto_lang_toolchain(
    name = "maven_java_toolchain",
    command_line = "--java_out=$(OUT)",
    runtime = "//dependencies/maven/artifacts/com/google/protobuf:protobuf-java",
    visibility = ["//visibility:public"],
)

Then, create a file named .bazelrc next to your WORKSPACE and add the following:

build --proto_toolchain_for_java=//:maven_java_toolchain

Then, the solution from my first comment should work as expected.

Note: I really recommend you to go with option 3. Otherwise, there's a chance you use two different versions of protobuf in your build, which can lead to weird and hard-to-debug failures (at built- and/or run-time).

[amit-traiana]

Thanks for the quick reply and the multiple solutions, I appreciate it :-)

Option 3 sounds the best for me too, but I'm afraid I am forced to use java_library because how pom_file implement dependencies.I tried to avoid complex example to make things simple, but it seems there is not way to avoid it :-)

# SERVICE A BUILD FILE

proto_library(
    name = "service_a_proto",
    srcs = [
        "service.proto",
    ],
    visibility = ["//visibility:public"],
)

java_proto_library(
    name = "service_a_java_proto",
    deps = [":service_a_proto"],
)

java_library(
    name = "service_a_java",
    srcs = [":service_a_java_proto"],
    deps = ["//dependencies/maven/artifacts/com/google/protobuf:protobuf-java"],
    visibility = ["//visibility:public"],
)

pom_file(
    name = "pom",
    template_file = "//tools:pom-template.xml",
    targets = [":service_a_java"],
)

# SERVICE B BUILD FILE (DEPENDS ON SERVICE A!)

proto_library(
    name = "service_b_proto",
    srcs = [
        "service_b.proto",
    ],
    deps = [ "//service_a:service_a_proto"],
    visibility = ["//visibility:public"],
)

java_proto_library(
    name = "service_b_java_proto",
    deps = [":service_b_proto"],
)

java_library(
    name = "service_b_java",
    srcs = [":service_b_java_proto"],
    deps = ["//service_a:service_a_proto",
                 "//dependencies/maven/artifacts/com/google/protobuf:protobuf-java"
     ],
    visibility = ["//visibility:public"],
)

pom_file(
    name = "pom",
    template_file = "//tools:pom-template.xml",
    targets = [":service_b_java"],
)

When running the above code, the pom file for service_b will include protobuf-java because it's a maven dependency, but it has no idea how to handle the Bazel proto dependency //service_a:service_a_proto, because that's not a real Maven dependency. So it will do nothing, and only the protobuf-java will be showing on the pom file. In order to fix it, you have to supply a tag inside the service_a_java definition like so:

java_library(
    name = "service_a_java",
    srcs = [":service_a_java_proto"],
    tags = ["maven_coordinates=bla:a_service_a:1.1"],
    deps = ["//dependencies/maven/artifacts/com/google/protobuf:protobuf-java"],
    visibility = ["//visibility:public"],
)

pom_file reads the tag, and create a dependency from this string (group id, artifact id and version) like so:

<groupId>bla</groupId>
<artifactId>a_service</artifactiId>
<version>1.1</version>

So to sum is up, I HAVE to use java_library, because If I to create Bazel dependencies inside the POM file, I have to supply a tag attribute for it to parse.

Another problem with the 3rd options is that we have multiple files of protobuf. Some are just protobuf, and in that case protobuf-java is enough. Some are of type gRPC protobuf - which means I need 3 more maven dependencies, and some even has Scala annotation so I need to add a dependency to the Scala proto implementation (scalaPb). It means I have to list them all inside the toolchain, but in reality only some of the JARs will be using all dependencies.

So correct me If I'm wrong, but only the 2nd option supplied will work due that limitation? But how can It be done without 'src'? java_library won't work with just deps, It also need src (and since I don't compile, I don't really have a src?).

I also agree what your saying about the problem with @com_google_protobuf. I wasn't thinking about the fact this is coming from a specific git commit, and the Maven use a other version. This indeed can cause issues. The only way I can see around it is to make sure the git commit pointing to @com_google_protobuf is the version I state on maven dependencies, but that's quite hard.

There still an open bug to support Mavel/Pom in Bazel here. Maybe I should link that issue and ask there also. I would love to hear more ideas now that you have a bigger picture of what's happening.

Also, @ronshapiro added the support in pom_file. Ron, I would love to hear you opinion about the problem if you will have the time :-)

Thank you!

[ceronman]

I'm not sure if you are still having this problem. But here is how I solve it.

You want to consume use the output of java_proto_library as sources (i.e using them from the srcs attribute instead of the deps attribute). But the problem is that java_library only understands sources that are either *.java or *.srcjar files. java_proto_library is able to generate source files, but these come as a *-src.jar file. So what I did was to have a small genrule that copies the *-src.jar to a *.srcjar. It's not elegant, but it works.

Here is your example adapted:

proto_library(
    name = "service_a_proto",
    srcs = [
        "service.proto",
    ],
    visibility = ["//visibility:public"],
)

java_proto_library(
    name = "service_a_java_proto",
    deps = [":service_a_proto"],
)

genrule(
    name = "service_a_java_proto_sources",
    srcs = [":service_a_java_proto"],
    outs = ["service_a_java_proto.srcjar"],
    cmd = "cp $(RULEDIR)/service_a_java_proto-speed-src.jar $(location service_a_java_proto.srcjar)",
)

java_library(
    name = "service_a_java",
    srcs = [":service_a_java_proto"],
    deps = ["//dependencies/maven/artifacts/com/google/protobuf:protobuf-java"],
    visibility = ["//visibility:public"],
)

Let me know if you have found a better solution.

Hopefully, java_library could be made a bit smarter so that it also understands *-src.jar files.

[amit-traiana]

Thanks for sharing your solution @ceronman ! What I ended up doing, is just using rules_proto_grpc, which wraps proto/grpc implementation across multiple languages. But I'll be using this trick for smaller projects I maintain, thanks!

[jkaye2012]

Another option for anyone else having this issue is to use java_export to generate jars by relying upon the sources as runtime_deps. Something akin to this is working well for us:

proto_library(
    name = "proto_lib",
    srcs = glob(["*.proto"]),
    visibility = ["//visibility:public"],
)

java_proto_library(
    name = "java_proto_lib",
    visibility = ["//visibility:public"],
    deps = [
        ":proto_lib",
    ],
)

java_export(
    name = "java_proto_export",
    maven_coordinates = "group:artifact:version",
    visibility = ["//visibility:public"],
    runtime_deps = [":java_proto_lib"],
)

A nice bonus of this is that downstream java_library rules can rely directly on the java_proto_export target as a standard dep.

[Sam-sad-Sajid]

@amit-traiana thanks a lot for your detailed walkthrough on the problem. It helped me a lot. I have some questions.

As you mentioned you used rules_proto_grpc, how did you then generate the pom file? I was trying to pipe the output like the following:

load("@rules_proto_grpc//java:defs.bzl", "java_grpc_library")
load("@rules_proto//proto:defs.bzl", "proto_library")
load("@google_bazel_common//tools/maven:pom_file.bzl", "pom_file")

proto_library(
    name = "a",
    srcs = [
        "a.proto"
    ],
    deps = ["@com_google_protobuf//:any_proto"],
    visibility = ["//visibility:public"]
)

java_grpc_library(
    name = "a_grpc",
    protos = [":a"],
    deps = [],
    visibility = ["//visibility:public"],
)

pom_file(
    name = "pom",
    template_file = "pom-template.xml",
    targets = [":a_grpc"],
)

But the generated pom file did not have google protobuf dependency :( Wondering how did you solve it?

And the last question is, you generated the jar files and the pom files, then how did you upload them together to an artifact repository? So that the artifact can be downloaded by another java project.

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions[bot]]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please post @bazelbuild/triage in a comment here and we'll take a look. Thanks!