ability to silence warnings when using multiple repositories

[joprice]

When using multiple repositories, the assumption is that the repositories will be iterated in order and a dependency will be resolved from the first that succeeds. Failing to find a dependency in any given repository is not usually of interest if it is found in at least one. When dependency resolution fails, however, it is convenient to have this information to recreate each successive resolution attempt. Silencing these warnings in the positive case, if possible, would be helpful, but perhaps the call doesn't need to be made at all: I see multiple urls for "mirror_urls" in my pin file, where, for any given dependency, I expect only one to work. If the dependency was resolved once from a given mirror_url, could that information be stored, and that url be tried first?

An example of the error message I am referring to:

WARNING: Download from ... failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found

[jin]

Yes, this is pretty annoying. I'd expect this feature to be wired up to repository_ctx.download, and not a rules_jvm_external specific feature though.

[Zetten]

Suppressing the warning should be handled in repository_ctx.download, I agree.

But some options to actually do some dependency management are applicable here - essentially I think it's not a safe assumption that every artifact exists in every repository in a given maven_install target.

In my particular case I have a large set of artifacts from Maven Central, but also a few artifacts from another repository. I could create two maven_install targets, but it leads to a potential issue as the transitive deps of the second artifact set may not be pinned to the versions I request from Central, plus developers have to figure out which workspace contains which artifact.

Could rules_jvm_external avoid producing mirror_urls for entries in maven_install.json? This could be switched either globally or per-artifact, or best (and slowest) of all, based on testing each artifact in each repository and emitting only known-good URLs.

Perhaps the most elegant option would be to allow composing several maven_install targets into a single external workspace/maven_install.json - so we could specify repositories+artifacts in known-good combinations, but still benefit from a single @maven//... set of targets.

[steeve]

How come the the old aar_ and maven_ repo rules didn't suffer from this problem?

[jin]

Could rules_jvm_external avoid producing mirror_urls for entries in maven_install.json? This could be switched either globally or per-artifact, or best (and slowest) of all, based on testing each artifact in each repository and emitting only known-good URLs.

emitting the known-good URLs would be my preferred choice, maybe just checking if there's a HTTP 200 on the http response headers to reduce the overhead.

How come the the old aar_ and maven_ repo rules didn't suffer from this problem?

The r_j_e implementation is naive -- all repositories are added to the urls list for every artifact, regardless of whether the artifact exists on the repository or not.

[Zetten]

emitting the known-good URLs would be my preferred choice, maybe just checking if there's a HTTP 200 on the http response headers to reduce the overhead.

I agree, this is the approach I've taken in bazel-dependencies-gradle-plugin - HEAD each artifact (including sources jars) from each repository and if you get a 200-399 code, treat it as a valid source for the artifact.

The plugin can also emit a maven_install.json compatible with rules_jvm_external, so it's essentially just running the pinning step via another tool. But we end up with only safe URLs, and have the advantages of Gradle's dependency management flexibility (including e.g. Spring's dependency-management-plugin).

[steeve]

The r_j_e implementation is naive -- all repositories are added to the urls list for every artifact, regardless of whether the artifact exists on the repository or not.

Also the warning may be sort of new. I see our old code (before rules_jvm_external) used to use ctx.download + java_import/aar_import with all urls, and I don't remember seeing as warning at the time. That was Feb 2019.

[alexeagle]

Hey @jin should we file an issue in bazelbuild/bazel on the repository_ctx.download function? Or do you think experimental_downloader_config is a suitable place for the 404 suppression to go?

[shs96c]

It's something that repository_ctx.download should handle. All the downloader config does is set up rules for rewriting URLs

[thirtyseven]

#743 fixed this, but it looks like the change to the V2 lockfile format introduced a regression.