Merge strategy does not discard duplicates when assembling multi-modules

[nextdtinc]

steps

1. unzip the attached minimal reproduction test case and run sbt assembly.

my-app.zip

The minimal test case contains only a single external library dependency, "com.amazonaws" % "aws-java-sdk" % "1.11.500", which contains duplicates of the file META-INF/io.netty.versions.properties, and this breaks the assembly.

problem

the assembly fails with this error:

[error] 1 error was encountered during merge
[info] Strategy 'discard' was applied to 376 files (Run the task at debug level to see details)
[info] Strategy 'filterDistinctLines' was applied to a file (Run the task at debug level to see details)
[info] Assembly up to date: target/batch-assembly-1.0-SNAPSHOT.jar

[error] java.lang.RuntimeException: deduplicate: different file contents found in the following:
[error] ~/.ivy2/cache/io.netty/netty-codec-http/jars/netty-codec-http-4.1.17.Final.jar:META-INF/io.netty.versions.properties
[error] ~/.ivy2/cache/io.netty/netty-codec/jars/netty-codec-4.1.17.Final.jar:META-INF/io.netty.versions.properties
[error] ~/.ivy2/cache/io.netty/netty-transport/jars/netty-transport-4.1.17.Final.jar:META-INF/io.netty.versions.properties
[error] ~/.ivy2/cache/io.netty/netty-buffer/jars/netty-buffer-4.1.17.Final.jar:META-INF/io.netty.versions.properties
[error] ~/.ivy2/cache/io.netty/netty-common/jars/netty-common-4.1.17.Final.jar:META-INF/io.netty.versions.properties
[error] ~/.ivy2/cache/io.netty/netty-resolver/jars/netty-resolver-4.1.17.Final.jar:META-INF/io.netty.versions.properties
[error] ~/.ivy2/cache/io.netty/netty-handler/jars/netty-handler-4.1.17.Final.jar:META-INF/io.netty.versions.properties

expectation

The assembly should have been successful, and the merge strategy should have discarded the duplicate file META-INF/io.netty.versions.properties.

Two different ways to accomplish this were attempted, none of them work:

assemblyMergeStrategy in assembly := {
 case PathList("META-INF", "io.netty.versions.properties") => MergeStrategy.discard
 case x if x.endsWith("META-INF/io.netty.versions.properties") => MergeStrategy.discard
 case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

notes

It works when the external dependency is replaced with an earlier version of the aws library, "com.amazonaws" % "aws-java-sdk" % "1.11.80", because that one does not contain any duplicates of META-INF/io.netty.versions.properties.

[dmigo]

It works when the external dependency is replaced with an earlier version of the aws library, "com.amazonaws" % "aws-java-sdk" % "1.11.80", because that one does not contain any duplicates of META-INF/io.netty.versions.properties.

Thank you for this part. This is at least a working workaround.

[pascals-ager]

Is there a work around to this problem with more recent versions of aws-sdk. I have similar issues with 2.17.99; None of the typical strategies seem to work. At this point its a lot of trial and error.

[NicolasBielza]

Using SBT 1.5.8 and aws-sdk 2.17.269, the following assemblyMergeStrategy does work:

  assembly / assemblyMergeStrategy := {
    case PathList("META-INF", "io.netty.versions.properties") => MergeStrategy.concat
    case PathList("META-INF", "versions", "9", "module-info.class") => MergeStrategy.discard
    case x => defaultTritonMergeStrategy(x) // Custom default strategy
  }