Issue with scala_specs2_unit_test + --strict_java_deps=ERROR

[jjudd]

Hi everyone!

When I build some of our scala_specs2_junit_test targets using --strict_java_deps=ERROR, I get errors that I'm not sure how to resolve.

error: Target 'Unknown label of file bazel-out/local-fastbuild/genfiles/external/scala/_ijar/scala-parser-combinators/external/scala/lib/scala-parser-combinators_2.11-1.0.4-ijar.jar which came from @io_bazel_rules_scala//specs2:specs2' is used but isn't explicitly declared, please add it to the deps.
You can use the following buildozer command:
buildozer 'add deps Unknown label of file bazel-out/local-fastbuild/genfiles/external/scala/_ijar/scala-parser-combinators/external/scala/lib/scala-parser-combinators_2.11-1.0.4-ijar.jar which came from @io_bazel_rules_scala//specs2:specs2' //util:test

error: Target 'Unknown label of file bazel-out/local-fastbuild/genfiles/external/scala/_ijar/scala-xml/external/scala/lib/scala-xml_2.11-1.0.5-ijar.jar which came from @io_bazel_rules_scala//specs2:specs2' is used but isn't explicitly declared, please add it to the deps.
You can use the following buildozer command:
buildozer 'add deps Unknown label of file bazel-out/local-fastbuild/genfiles/external/scala/_ijar/scala-xml/external/scala/lib/scala-xml_2.11-1.0.5-ijar.jar which came from @io_bazel_rules_scala//specs2:specs2' //util:test
two errors found
two errors found

Normally I would add the jar to the deps list, but I'm not sure how to do that here. Both scala-parser-combinators and scala-xml are maven_jars in my WORKSPACE file. I've added them to the deps list for the test target, but it doesn't seem to change anything.

If there's a bug, I'm happy to provide debug info and fix it. If this is an issue with my ignorance of bazel, then I'm happy to update docs somewhere to help others avoid this in the future.

Here are the rules from the BUILD file

shared_deps = [
  "//a",
  "//b",
  "//c",
  "//d",
  "@com_typesafe_scala_logging_scala_logging_2_11//jar:file",
  "@com_fasterxml_jackson_core_jackson_annotations//jar",
  "@com_fasterxml_jackson_core_jackson_core//jar",
  "@com_fasterxml_jackson_core_jackson_databind//jar",
  "@com_typesafe_config//jar",
  "@com_typesafe_play_play_datacommons_2_11//jar",
  "@com_typesafe_play_play_functional_2_11//jar",
  "@com_typesafe_play_play_iteratees_2_11//jar",
  "@com_typesafe_play_play_json_2_11//jar",
  "@commons_io_commons_io//jar",
  "@joda_time_joda_time//jar",
  "@org_apache_httpcomponents_httpclient//jar",
  "@org_apache_httpcomponents_httpcore//jar",
  "@org_apache_httpcomponents_httpmime//jar",
  "@org_scala_lang_modules_scala_parser_combinators_2_11//jar",
  "@org_scala_lang_modules_scala_xml_2_11//jar",
  "@org_scala_lang_plugins_scala_continuations_library_2_11//jar",
  "@org_slf4j_slf4j_api//jar",
]

scala_macro_library(
  name = "util",
  srcs = glob(["src/main/**/*.scala"]) + glob(["src/main/**/*.java"]),
  resources = glob(["src/main/resources/**"]),
  visibility = ["//visibility:public"],
  deps = shared_deps + [
    "//e",
    "//f",
    "//g",
    "//e",
    "@com_amazonaws_aws_java_sdk_core//jar",
    "@com_amazonaws_aws_java_sdk_sts//jar",
    "@com_github_nscala_time_nscala_time_2_11//jar",
    "@com_lucidchart_thread_context//jar",
    "@commons_codec_commons_codec//jar",
    "@net_sf_ehcache_ehcache_core//jar",
    "@org_apache_httpcomponents_httpasyncclient//jar",
    "@org_scala_lang_modules_scala_java8_compat_2_11//jar",
    "@org_scalaz_scalaz_core_2_11//jar:file",
  ]
)

scala_specs2_junit_test(
  name = "test",
  srcs = glob(["src/test/**/*.scala"]),
  resources = glob(["src/test/resources/**"]),
  deps = shared_deps + [
    ":util",
      "@junit_junit//jar",
      "@org_mockito_mockito_core//jar",
      "@org_objenesis_objenesis//jar",
      "@org_specs2_specs2_mock_2_11//jar",
  ],
  suffixes = ["Spec", "Test"],
  timeout = "short",
  print_discovered_classes = True,
)

[natansil]

Maybe this commit surfaced a bug (not sure where), as there is one more level of dependencies...

@ittaiz WDYT?

[ittaiz]

For strict deps to work well we need to know the targets of the transitive dependencies and that isn't currently supported in JavaInfo (the java common provider). To that end we (rules_scala) pass that information along transitively. If you use any rule which isn't rules_scala (or scala_import) you will get strange labels which mainly tell you which jar caused the problem and now you the user need to find the actual label. scala_import which I mention above is my own attempt for java_import but one which amongst other things supports this label propagation. We use it internally and we have very little label problems.

Problems with what I'm telling you:

1. scala_import is still in an unbaked PR and not in master
2. If you have java_library targets you'll stumble on to this as well
3. Even if all will be well there is still an edge case with how java rules evaluate the labels and we work around that with a script that takes the "close enough label" and massages it to be the correct label.

Hopefully I'll have time soon to work on 1 (if you want to add tests to it you're more than welcome). Re 2, I hope @iirina from Bazel team will have some time for us to revamp again the JavaInfo area. Re 3, again maybe @iirina will be able to assist us but it's lower on priority.

[natansil]

@ittaiz, in this specific case, is the 'problematic' rulejava_import (specs2 target)?

[ittaiz]

java_import is also unsupported due to the problem mentioned above. I think this also surfaces another problem where we warn on transitive issues of libraries which we should consider whitelisting like scala parser combinators.

[natansil]

@jjudd there are two issues here:

1. missing labels - related to use of java rules which do not yet support label propagation interop with scala rules
2. error in missing direct deps - still unknown cause.

which versions are you using for @org_scala_lang_modules_scala_parser_combinators_2_11 and @org_scala_lang_modules_scala_xml_2_11? maybe they are different...

[natansil]

by different I mean that maybe the versions @org_scala_lang_modules_scala_parser_combinators_2_11 and @org_scala_lang_modules_scala_xml_2_11 are different than the versions of jars that are contained in scala_repositories brought by rules_scala

[ittaiz]

@jjudd any updates? We'd love to help you; Just need some more info

[jjudd]

Sorry for the lack of updates. I have an afternoon to sit down and look at this today. I'll let you know what I find.

[ittaiz]

Sure thing. Good luck!

On Sat, Nov 18, 2017 at 9:37 PM James Judd [email protected] wrote:

Sorry for the lack of updates. I have an afternoon to sit down and look at this today. I'll let you know what I find.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/rules_scala/issues/338#issuecomment-345465795, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUIF3-U8wtx970FTMrtNQ6fTarFd44qks5s3zISgaJpZM4Qd1Qo .

[jjudd]

I was able to work around this by replacing everywhere I was using @org_scala_lang_modules_scala_xml_2_11//jar and @org_scala_lang_modules_scala_parser_combinators_2_11//jar with @scala//:scala-xml and @scala//:scala-parser-combinators.

We can now build everything with --strict_java_deps=ERROR. :tada: Thanks for your help.

I also tried changing the version of the maven_jar to the same version as defined in rules_scala, but that didn't seem to have an impact. These were the versions in my WORKSPACE when I started:

  name = "org_scala_lang_modules_scala_xml_2_11",
  artifact = "org.scala-lang.modules:scala-xml_2.11:1.0.1",
  sha1 = "21dbac0088b91b8ffb7ac385301f4340f8ebe71f",
)

maven_jar(
  name = "org_scala_lang_modules_scala_parser_combinators_2_11",
  artifact = "org.scala-lang.modules:scala-parser-combinators_2.11:1.0.5",
  sha1 = "cbd78079c99262f7a535d12a00a2dc3da6a266a0",
)

Compare that to the rules_scala versions.

java_import(
    name = "scala-xml",
    jars = ["lib/scala-xml_2.11-1.0.5.jar"],
    visibility = ["//visibility:public"],
)
java_import(
    name = "scala-parser-combinators",
    jars = ["lib/scala-parser-combinators_2.11-1.0.4.jar"],
    visibility = ["//visibility:public"],

The workaround is alright, but is it possible to use the specs2 rules and depend on a different version of scala-xml and scala-parser-combinators than the versions in rules_scala? I can imagine not everyone will want their xml and parser-combinator libraries to be in lockstep with the rules_scala versions.

I'm still pretty new to Bazel, so I'm not terribly familiar with java_import vs scala_import and things like that. Thanks for the info, it's helpful.

[ittaiz]

Yes you can. The solution is to not load scala_repositories and instead bind all of the labels that methods binds to targets you've defined. Re what you mentioned- indeed the versions might be the same but in bazel the label is king and you added different dependencies as far as bazel cares (i.e. it doesn't peak into the jar/maven-coordinates)

On Sun, Nov 19, 2017 at 10:37 PM James Judd [email protected] wrote:

I was able to work around this by replacing everywhere I was using @org_scala_lang_modules_scala_xml_2_11//jar and @org_scala_lang_modules_scala_parser_combinators_2_11//jar with @scala//:scala-xml and @scala//:scala-parser-combinators.

We can now build everything with --strict_java_deps=ERROR. 🎉 Thanks for your help.

I also tried changing the version of the maven_jar to the same version as defined in rules_scala, but that didn't seem to have an impact. These were the versions in my WORKSPACE when I started:

name = "org_scala_lang_modules_scala_xml_2_11", artifact = "org.scala-lang.modules:scala-xml_2.11:1.0.1", sha1 = "21dbac0088b91b8ffb7ac385301f4340f8ebe71f", )

maven_jar( name = "org_scala_lang_modules_scala_parser_combinators_2_11", artifact = "org.scala-lang.modules:scala-parser-combinators_2.11:1.0.5", sha1 = "cbd78079c99262f7a535d12a00a2dc3da6a266a0", )

Compare that to the rules_scala versions.

java_import( name = "scala-xml", jars = ["lib/scala-xml_2.11-1.0.5.jar"], visibility = ["//visibility:public"], ) java_import( name = "scala-parser-combinators", jars = ["lib/scala-parser-combinators_2.11-1.0.4.jar"], visibility = ["//visibility:public"],

The workaround is alright, but is it possible to use the specs2 rules and depend on a different version of scala-xml and scala-parser-combinators than the versions in rules_scala? I can imagine not everyone will want their xml and parser-combinator libraries to be in lockstep with the rules_scala versions.

I'm still pretty new to Bazel, so I'm not terribly familiar with java_import vs scala_import and things like that. Thanks for the info, it's helpful.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/rules_scala/issues/338#issuecomment-345547577, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUIF8SD0XVch6PyoMqEvv987XpStdF8ks5s4JGKgaJpZM4Qd1Qo .