rules_jvm_external

rules_jvm_external copied to clipboard

Importing io.netty:netty-tcnative-boringssl-static:2.0.52.Final makes bazel complain about cyclic error

$ bazel build @maven//:io_netty_netty_tcnative_boringssl_static
.../external/maven/BUILD:84:11: in jvm_import rule @maven//:io_netty_netty_tcnative_boringssl_static_windows_x86_64: cycle in dependency graph:
    @maven//:io_netty_netty_tcnative_boringssl_static (d5a9994f7dbed333fedf2517edff904e2ccf8dfa4714a908e2c4920ae526e339)
.-> @maven//:io_netty_netty_tcnative_boringssl_static_windows_x86_64 (d5a9994f7dbed333fedf2517edff904e2ccf8dfa4714a908e2c4920ae526e339)
|   @maven//:io_netty_netty_tcnative_boringssl_static_linux_x86_64 (d5a9994f7dbed333fedf2517edff904e2ccf8dfa4714a908e2c4920ae526e339)
`-- @maven//:io_netty_netty_tcnative_boringssl_static_windows_x86_64 (d5a9994f7dbed333fedf2517edff904e2ccf8dfa4714a908e2c4920ae526e339)

So all architectures try to depend on all architectures which creates cycles.

$ bazel query --output build @maven//:io_netty_netty_tcnative_boringssl_static_windows_x86_64

jvm_import(
  name = "io_netty_netty_tcnative_boringssl_static_windows_x86_64",
  tags = ["maven_coordinates=io.netty:netty-tcnative-boringssl-static:jar:windows-x86_64:2.0.52.Final", "maven_url=https://repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.52.Final/netty-tcnative-boringssl-static-2.0.52.Final-windows-x86_64.jar"],
  jars = ["@maven//:v1/https/repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.52.Final/netty-tcnative-boringssl-static-2.0.52.Final-windows-x86_64.jar"],
  deps = ["@maven//:io_netty_netty_tcnative_boringssl_static_linux_x86_64", "@maven//:io_netty_netty_tcnative_boringssl_static_osx_aarch_64", "@maven//:io_netty_netty_tcnative_boringssl_static_osx_x86_64", "@maven//:io_netty_netty_tcnative_classes", "@maven//:io_netty_netty_tcnative_boringssl_static_linux_aarch_64"],
)

$ cat "$(bazel info output_base)/external/maven/dep-tree.json  | jq ."
...
    {
      "coord": "io.netty:netty-tcnative-boringssl-static:jar:windows-x86_64:2.0.52.Final",
      "file": "v1/https/repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.52.Final/netty-tcnative-boringssl-static-2.0.52.Final-windows-x86_64.jar",
      "directDependencies": [
        "io.netty:netty-tcnative-boringssl-static:jar:linux-x86_64:2.0.52.Final",
        "io.netty:netty-tcnative-boringssl-static:jar:osx-aarch_64:2.0.52.Final",
        "io.netty:netty-tcnative-boringssl-static:jar:osx-x86_64:2.0.52.Final",
        "io.netty:netty-tcnative-classes:2.0.52.Final",
        "io.netty:netty-tcnative-boringssl-static:jar:windows-x86_64:2.0.52.Final",
        "io.netty:netty-tcnative-boringssl-static:jar:linux-aarch_64:2.0.52.Final"
      ],
...

POM and artifacts

https://repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.52.Final/netty-tcnative-boringssl-static-2.0.52.Final.pom https://repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.52.Final/

Setup

WORKSPACE

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

RULES_JVM_EXTERNAL_TAG = "4.2"
RULES_JVM_EXTERNAL_SHA = "cd1a77b7b02e8e008439ca76fd34f5b07aecb8c752961f9640dea15e9e5ba1ca"

http_archive(
    name = "rules_jvm_external",
    strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG,
    sha256 = RULES_JVM_EXTERNAL_SHA,
    url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG,
)

load("@rules_jvm_external//:repositories.bzl", "rules_jvm_external_deps")

rules_jvm_external_deps()

load("@rules_jvm_external//:setup.bzl", "rules_jvm_external_setup")

rules_jvm_external_setup()

load("@rules_jvm_external//:defs.bzl", "maven_install")

maven_install(
    artifacts = [
        "io.netty:netty-tcnative-boringssl-static:2.0.52.Final",
    ],
    repositories = [
        "https://repo1.maven.org/maven2",
    ],
)

BUILD - empty file

Root cause

Not really sure if this indicates a malformed POM that indeed defines cycles (also not sure if they are illegal with maven). Or could be coursier or rules_jvm_external getting confused by default classifier depending on per-platform classifier artifacts.

As a workaround override_targets probably can help

The POM looks weird, but I can see why they did that. Agree that override_targets would probably be the best workaround here, short of making coursier/rules_jvm_external play nice with this POM.

@jin @dmivankov do you have an example for how override_targets can be used in this situation?

I also encountered the same problem: @dmivankov @jin , how to resolve this problem.

ERROR: /private/var/tmp/_bazel_shaoxixu/241205fdee0dfc3430ee79ef9f505ed0/external/maven/BUILD:3171:11: in jvm_import rule @maven//:io_netty_netty_tcnative_boringssl_static_linux_aarch_64: cycle in dependency graph:
    //cover:cover_appjar_locator (b3d8f083ae5ab0b8cd3bafc91962d5d549a5a65e3e548f133ac94506b7efecb4)
    //cover:cover_lib (b3d8f083ae5ab0b8cd3bafc91962d5d549a5a65e3e548f133ac94506b7efecb4)
    @maven//:net_devh_grpc_server_spring_boot_starter (b3d8f083ae5ab0b8cd3bafc91962d5d549a5a65e3e548f133ac94506b7efecb4)
    @maven//:net_devh_grpc_server_spring_boot_autoconfigure (b3d8f083ae5ab0b8cd3bafc91962d5d549a5a65e3e548f133ac94506b7efecb4)
    @io_grpc_grpc_java//netty:shaded_maven (b3d8f083ae5ab0b8cd3bafc91962d5d549a5a65e3e548f133ac94506b7efecb4)
    @io_grpc_grpc_java//netty/shaded:shaded (b3d8f083ae5ab0b8cd3bafc91962d5d549a5a65e3e548f133ac94506b7efecb4)
    @io_netty_netty_tcnative_boringssl_static//jar:jar (b3d8f083ae5ab0b8cd3bafc91962d5d549a5a65e3e548f133ac94506b7efecb4)
    @maven//:io_netty_netty_tcnative_boringssl_static (b3d8f083ae5ab0b8cd3bafc91962d5d549a5a65e3e548f133ac94506b7efecb4)
.-> @maven//:io_netty_netty_tcnative_boringssl_static_linux_aarch_64 (b3d8f083ae5ab0b8cd3bafc91962d5d549a5a65e3e548f133ac94506b7efecb4)
|   @maven//:io_netty_netty_tcnative_boringssl_static_osx_aarch_64 (b3d8f083ae5ab0b8cd3bafc91962d5d549a5a65e3e548f133ac94506b7efecb4)
`-- @maven//:io_netty_netty_tcnative_boringssl_static_linux_aarch_64 (b3d8f083ae5ab0b8cd3bafc91962d5d549a5a65e3e548f133ac94506b7efecb4)
ERROR: Analysis of target '//cover:cover_appjar_locator' failed; build aborted
INFO: Elapsed time: 0.218s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (24 packages loaded, 780 targets configured)

Workaround is to manually fetch cyclic dependencies and override them, like so

    override_targets = {
        "io.netty:netty-tcnative-boringssl-static": "//workaround:io_netty_netty_tcnative_boringssl_static",
        "io.netty:netty-tcnative-boringssl-static:osx-x86_64": "//workaround:io_netty_netty_tcnative_boringssl_static_osx_x86_64",
        "io.netty:netty-tcnative-boringssl-static:osx-aarch_64": "//workaround:io_netty_netty_tcnative_boringssl_static_osx_aarch_64",
        "io.netty:netty-tcnative-boringssl-static:windows-x86_64": "//workaround:io_netty_netty_tcnative_boringssl_static_windows_x86_64",
        "io.netty:netty-tcnative-boringssl-static:linux-x86_64": "//workaround:io_netty_netty_tcnative_boringssl_static_linux_x86_64",
        "io.netty:netty-tcnative-boringssl-static:linux-aarch_64": "//workaround:io_netty_netty_tcnative_boringssl_static_linux_aarch_64",
    },

and use definitions from external/maven/BUILD, but modify deps to remove cycles and also substitute @maven with dependencies/ workaround/BUILD would look similar to

java_import(
    name = "io_netty_netty_tcnative_boringssl_static_osx_x86_64",
    jars = ["@maven//:v1/https/..."],
    tags = ["maven_coordinates=io.netty:netty-tcnative-boringssl-static:jar:osx-x86_64:2.0.51.Final"],
)

java_import(
    name = "io_netty_netty_tcnative_boringssl_static_osx_aarch_64",
    jars = ["@maven//:v1/https/..."],
    tags = ["maven_coordinates=io.netty:netty-tcnative-boringssl-static:jar:osx-aarch_64:2.0.51.Final"],
)

java_import(
    name = "io_netty_netty_tcnative_boringssl_static_linux_x86_64",
    jars = ["@maven//:v1/https/..."],
    tags = ["maven_coordinates=io.netty:netty-tcnative-boringssl-static:jar:linux-x86_64:2.0.51.Final"],
)

java_import(
    name = "io_netty_netty_tcnative_boringssl_static_linux_aarch_64",
    jars = ["@maven//:v1/https/..."],
    tags = ["maven_coordinates=io.netty:netty-tcnative-boringssl-static:jar:linux-aarch_64:2.0.51.Final"],
)

java_import(
    name = "io_netty_netty_tcnative_boringssl_static_windows_x86_64",
    jars = ["@maven//:v1/https/..."],
    tags = ["maven_coordinates=io.netty:netty-tcnative-boringssl-static:jar:windows-x86_64:2.0.51.Final"],
)

java_import(
    name = "io_netty_netty_tcnative_boringssl_static",
    jars = ["@maven//:v1/https/..."],
    tags = [
        "maven_coordinates=io.netty:netty-tcnative-boringssl-static:2.0.51.Final",
        "maven_url=...",
    ],
    deps = [
        "//workaround:io_netty_netty_tcnative_boringssl_static_linux_aarch_64",
        "//workaround:io_netty_netty_tcnative_boringssl_static_linux_x86_64",
        "//workaround:io_netty_netty_tcnative_boringssl_static_osx_aarch_64",
        "//workaround:io_netty_netty_tcnative_boringssl_static_osx_x86_64",
        "//workaround:io_netty_netty_tcnative_boringssl_static_windows_x86_64",
        "@maven//:io_netty_netty_tcnative_classes",
    ],
)

code snippets above are only for illustration and need tuning to your repo: maven urls, versions, location of workaround BUILD file etc

@dmivankov thanks for the prompt response, but I have some questions since my experience with bazel is surface level.

Workaround is to manually fetch cyclic dependencies

What's the process for fetching the dependencies manually? My understanding is that we can't rely on maven to do this and so it would require another mechanism.

and use definitions from external/maven/BUILD, but modify deps to remove cycles and also substitute @maven with dependencies/

Could you please clarify what you mean by this? Are you saying the the java_import's jars attribute needs to be changed to use dependencies/? Where is this dependencies/ entity defined?

Unfortunately that's not a simple workaround.

1. find dependencies that form a cycle (io.netty:netty-tcnative-boringssl-static* in this case)
2. copy their definitions to a new BUILD file (would look similar to workaround/BUILD listed above)
3. modify BUILD file to remove cycles and use those newly added targets for cross-dependencies between them (deps= of io_netty_netty_tcnative_boringssl_static java_import should use //workaround:io_netty_netty_tcnative_boringssl_static* instead of @maven//:io_netty_netty_tcnative_boringssl_static*)
4. add override_targets = {...} to maven_install to override maven coordinates to use targets from BUILD file (override_targets= example from comment above)

The part that was confusing to me was that your java_import rule's jars attribute mentioned the @maven repository. I don't believe that is possible since we are instructing maven to avoid these dependencies.

I was able to fix this by using http_jar and a full example can bee seen on https://github.com/pixie-io/pixie/pull/562.

I see, currently override_targets works with label generation after artifact resolution, so jars still remain available. http_jar approach is more future-proof :+1:

@dmivankov I believe this is a duplicate of https://github.com/bazelbuild/rules_jvm_external/issues/686 and after upgrading to rules_jvm_external 4.3 I no longer have this issue.

Looks like it, removed override_targets locally and rerun repin, seems to have worked :tada: