Wrong checksums on several (transitive) dependencies

[haikalpribadi]

Hi everyone,

I'm trying out to migrate our relatively large pom.xml at the moment, using coursier resolverType and I'm getting several checksum errors such as:

java.lang.RuntimeException: we could not download the artifact commons-httpclient:commons-httpclient:3.1 to compute the hash for digest type Sha1 with error com.github.johnynek.bazel_deps.CoursierResolver$FileErrorException: wrong checksum: /Users/haikalpribadi/Library/Caches/Coursier/v1/https/jcenter.bintray.com/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar
	at com.github.johnynek.bazel_deps.CoursierResolver$$anonfun$2.apply(CoursierResolver.scala:98)
        ...

I tried explicitly declaring commons-httpclient:commons-http-client:3.1 in input yaml file, and the error went away.

However, I keep on getting more of these errors. Do we know what the real cause is, and what is the best way to resolve these checksum errors?

This is the configuration of my dependency list (yaml input) file to bazel-deps:

options:
  languages: ["java"]
  resolverType: coursier
  resolvers:
  - id: mavencentral
    type: default
    url: "https://jcenter.bintray.com"
  strictVisibility: true
  transitivity: runtime_deps
  versionConflictPolicy: highest
  thirdPartyDirectory: "dependencies/maven/"

ps: I don't get these errors when resolverType: aether, but the documentation says "aether is the default, but it is slower and seems to silently miss some dependencies for reasons we don't yet understand" - and this makes me nervous when using it - are we able to trust it?

[johnynek]

so, you are downloading from bintray there...

Why is that? could they be silently changing the jars?

A difference with aether vs coursier is that coursier, tries to fetch the declared sha, it may be that the sha you downloaded does not match the sha when you fetch....

Maybe you should check manually if they do actually match?

cc @ianoc have we seen anything like this?

[haikalpribadi]

Thanks, @johnynek.

so, you are downloading from bintray there... Why is that? could they be silently changing the jars?

It produced less "missing" warnings/errors than the other repositories, and seem to be more complete.

it may be that the SHA you downloaded does not match the SHA when you fetch

How is the above scenario possible? And how do I avoid it?

A difference with aether vs coursier is that coursier, tries to fetch the declared sha

Does that mean if aether passes fine (ignoring the SHA checks for now), I should be able to trust the generated bazel dependencies will contain all the required artifacts to build and run the system?

Maybe you should check manually if they do actually match?

How do you do that?

[haikalpribadi]

At the moment, the reason why I can't move forward with the aether generated dependencies (even though it completes with no errors) is because of issue #184.

Do you know a workaround that I could for now?

[johnynek]

so, maven jars declare the sha, e.g.:

https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1/

see: https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar.sha1 964cd74171f427720480efdec40a7c7f6e58426a

So, coursier, fetches that value and verifies the hash. You can see if the URL you are downloading has that sha1 value.

If it does, there is some problem with coursier and/or bazel-deps. if it does not, that means that bintray is not transparently copying the file but is reprocessing it in some way.

This could happen if bintray recompresses the jars, or does any kind of transformation. You can check this using curl and shasum to check the values. Let's try to figure out the above so we can see where the issue is.

Also, you should be able to use maven central, so, could you also open an issue about the problems you see if you use maven central?

We use an internal nexus server that proxies to several remote maven servers, but I don't recall an issue exactly like this.

[haikalpribadi]

I've reverted back to url: https://repo.maven.apache.org/maven2/ (and resolverType: "coursier").

And now it complains:

[main] ERROR MakeDeps - resolution and sha collection failed
java.lang.RuntimeException: no artifacts for servletapi:servletapi:2.3 found
java.lang.RuntimeException: no artifacts for servletapi:servletapi:2.3 found
	at com.github.johnynek.bazel_deps.CoursierResolver$$anonfun$com$github$johnynek$bazel_deps$CoursierResolver$$lookup$1$1$$anonfun$apply$7$$anonfun$apply$9.apply(CoursierResolver.scala:188)
	at com.github.johnynek.bazel_deps.CoursierResolver$$anonfun$com$github$johnynek$bazel_deps$CoursierResolver$$lookup$1$1$$anonfun$apply$7$$anonfun$apply$9.apply(CoursierResolver.scala:186)
	at coursier.util.Task$$anonfun$flatMap$extension$1$$anonfun$apply$1.apply(Task.scala:10)
	at coursier.util.Task$$anonfun$flatMap$extension$1$$anonfun$apply$1.apply(Task.scala:10)
	at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
	at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
	at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

While we can see at maven central that servletapi:servletapi:2.3 exists: https://repo.maven.apache.org/maven2/servletapi/servletapi/2.3/

Any ideas?

And I suppose I should open a new issue for this too?

thanks @johnynek !

[haikalpribadi]

is there a config to disable the sha check for coursier, @johnynek ?

[johnynek]

interestingly, servletapi is not named as files normally are, note it is servletapi-2.3.jar not servletapi-servletap-2.3.jar

This may be a bug with coursier. We could try bumping to the latest coursier or seeing if we can reproduce the bug just using the coursier command line tool:

https://github.com/coursier/coursier#command-line

it looks like there is a later version of coursier if you want to open a PR to see if we can use the latest in this repo and see if it possibly unblocks you?

[haikalpribadi]

Sorry for the late reply on this, @johnynek. I will try to upgrade coursier once I'm done with our current migration from Maven to Bazel, and will let you know my findings here. For now, I've just removed it from our dependency.