sbt-pack Eliminate Duplicate Jars and Give a Useful Warning

In multi-project mode, if projectA depends on a-1.0.0.jar and projectB depends on a-1.0.1.jar, the pack plugin will copy both a-1.0.0.jar and a-1.0.1.jar which will cause a runtime problem. The feature is desired to have some strategy to eliminate duplicate jars with different versions that are brought in by different projects. The two strategy that came up to my mind is: 'latest' -> use the jar with higher version and print useful warning, 'exit' -> fail the task with useful message

Feb 24 '14 18:02 zhuchenwang

I would like to change the tuple (ClassPath, ProjectRef) into a case class, and add a documentation for this new feature.

As a reminder to myself, let me reopen this issue.

Mar 01 '14 07:03 xerial

"dependencyClasspath in Runtime" task on top level project should do the proper dependency resolution between submodules, I believe. The product of this task also contains classpath of dependent submodules like (Attributed(//target/scala-2.10/classes)), which should be possible to filter out and leave jar only in the array.

There is how sbt-package-dist does this job:

https://github.com/twitter/sbt-package-dist/blob/master/src/main/scala/com/twitter/sbt/PackageDist.scala#L245

Apr 07 '14 16:04 viktortnk

@viktortnk I think, the assumption for this argument is the top level project depends on all the sub-modules. In my case, the top level project is actually a dummy project and you can only see the scala library in the dependencyClasspath if you show it in sbt. If we assume there is always a module that will include all other modules as dependencies either directly or transitively, then "fullClasspath in Runtime" in that specific module will resolve the issue automatically.

Apr 07 '14 17:04 zhuchenwang

@zhuchenwang I didn't especially mean very top-level project. In my setup I have the similar dummy project which is "umbrella" (aggregates dependency type) for two others, intended to run separately. Those two projects share other submodules using "dependsOn" relationship. So I expected to specify "sbt-pack" settings for both of them

Sorry, I didn't consider your version of setup

Apr 07 '14 20:04 viktortnk

@viktortnk Never mind. Actually my test case https://github.com/xerial/sbt-pack/tree/develop/src/sbt-test/sbt-pack/duplicate-jars does report the warning for duplicate JARs in this setup. And the code seems to be using neither the dependencyClasspath nor fullClasspath. @xerial Is there any specific reason for using the update report instead of fullClasspath?

Apr 07 '14 23:04 zhuchenwang

@zhuchenwang sbt-pack need to use updateReport to obtain module information, which contains organization, name and revision values for each jar file. sbt-pack uses these data to rename jar file names if necessary.

Apr 10 '14 13:04 xerial

@xerial When it might be necessary to rename dependency jars?

Apr 10 '14 15:04 viktortnk

+1 Running into a problem in a multi project build where multiple versions of that same jar is packed, even though fullClasspath and dependencyClasspath shows the correct jars (single version of each), which is what I expect given a few careful excludes. This is a blocker for me since it could cause unexpected runtime issues :(

Apr 12 '14 00:04 nefilim

the duplicate/conflict resolution in 0.5.2-SNAPSHOT reduces the duplicates to a single jar but picks the newest jar, in my case I need the older (as reflected in dependencyClasspath)

Apr 12 '14 01:04 nefilim

@nefilim Short term solution, from the warning introduced in 0.5.2-SNAPSHOT, you can find out which module results in this conflict and add dependencyOverrides in that module to force it using an earlier version. @xerial The code needs to be revisited to figure out if we can leverage fullClasspath or dependencyClasspath.

Apr 12 '14 01:04 zhuchenwang

@zhuchenwang thanks for the suggestion, I tried it but it made no difference unfortunately, which I was expecting because I already had .force() on these dependencies.

Apr 12 '14 16:04 nefilim

It looks like classpaths generated from updateReports include duplicate jars, which cannot be found in dependencyClasspath. OK. sbt-pack should be fixed to check dependencyClasspath to remove duplicate jars.

I also noticed this problem, but it depends on the state of .ivy caches and project class files in project/target folder. So, it was hard to reproduce.

@viktortnk Renaming jars is necessary to cope with the other sbt plugins. For example, sbt-native-packager uses (orgnization-name)-(artifact name)-(version).jar pattern in default to package jar files.
https://github.com/xerial/sbt-pack/issues/18

Apr 15 '14 02:04 xerial