NoClassDefFoundError when shading

[zzvara]

I've been searching for related bugs and errors on shading with sbt-assembly, but solely a Windows-related bug have been found by me, which should not effect my build, due to running the latest version available. I'm trying to apply the shade rule:

ShadeRule .rename("javax.servlet.**" -> "javax.servlet-api.3.1.0.@1") .inLibrary("javax.servlet" % "javax.servlet-api" % "3.1.0").inProject

Which comes from Spark 2.2.0. The JAR is assembled on Windows and deployed to YARN running on Linux. The class is not found however, it seems that references are somehow not updated. Nothing, but my big fat JAR is submitted to YARN, no other Spark dependency comes from anywhere.

java.lang.NoClassDefFoundError: javax/servlet/FilterRegistration at org.eclipse.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:154) at org.eclipse.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:146) at org.eclipse.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:140) at org.eclipse.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:110) at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:135) at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:122) at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:81) at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:65) at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:65) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:65) at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:70) at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:86) at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:227) at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:169) at org.apache.spark.SparkContext.<init>(SparkContext.scala:446)

Is this a bug or I'm messing up something?

[manuzhang]

Isn't this expected since you've shaded the javax.servlet package to javax.servlet-api.3.1.0 ?

[zzvara]

Isn't the references (imports) to javax.servlet should be updated to javax.servlet-api.3.1.0 in the whole assembly, including Spark?

[manuzhang]

The error says "java.lang.NoClassDefFoundError: javax/servlet/FilterRegistration" so it's already updated.

[zzvara]

Doesn't this suggest that Spark is still looking for {{javax/servlet/FilterRegistration}} instead of {{javax/servlet-api/3/1/0/FilterRegistration}}? Please clarify.

[manuzhang]

I see. Then try inAll instead of inLibrary("javax.servlet" % "javax.servlet-api" % "3.1.0").inProject

[zzvara]

I tried that, and it worked, however, it only shaded the 3.5.0 that I use in my project. As I see 3.1.0 is used by Spark and is evicted by my 3.5.0. When I add a shade rule to shade 3.1.0, it is not applied, since that version is being evicted. Is there any way to solve this problem? This library seems to go against all rules, since I already have 30 shades rules in an enormous project, and those rules do work.

[zzvara]

I have further investigated the problem. Let's say that module X comes from project dependencies as follows: project dependency A depends on (pulls in) X version V1, project dependency B depends on X version V2. V1 and V2 not binary compatible, V1 > V2. SBT evicts one of the versions, V2 by default.

1. ShadeRule.rename(basePath.** -> shaded.V1.basePath.@1).inLibrary(X % V1) This rule will rewrite the library correctly, checking the assembly confirms this, but references are not updated in A. This leads to NoClassDefFoundError during runtime, when A tries to use something from X V1.
2. ShadeRule.rename(basePath.** -> shaded.V1.basePath.@1).inLibrary(X % V1).inProject Has exactly the same effect in this use case, than 1.
3. ShadeRule.rename(basePath.** -> shaded.V1.basePath.@1).inLibrary(X % V1).inProject.inAll This rule will update references as well in A, but incorrectly in B as well. When B incorrectly tries to call into X V1 instead of X V2.
4. ShadeRule.rename(basePath.** -> shaded.V1.basePath.@1).inAll Similar effect to 3.

Anyway, references are not updated correctly.

[manuzhang]

In your case, either A or B has to shade X itself (or you do it in a separate project). Otherwise, V2 has already been evicted before assembly comes in.

[zzvara]

@manuzhang Thanks, this is what I have been worried about. However, in this case, it would be feasible to "evict" X V2, shade X V1 and just "provide" X V2 runtime. The only problem is that none of the above shade rules update references correctly. Either both V1 and V2 references will be updated to the same rename rule, or none. I think this is either a bug, or a serious limitation that needs to be explicitly noted in README.

[manuzhang]

Have you tried ShadeRule.rename(basePath.** -> shaded.V1.basePath.@1).inLibrary(A, X % V1).inProject ?

[zzvara]

No. How do you define such a constraint on a ModuleID that inLibrary expects? def inLibrary(moduleId : sbt.ModuleID*) : sbtassembly.ShadeRule Example: X := org.json4s % json4s-jackson_2.11, V1 := 3.5.3, V2 := 3.2.11.

[manuzhang]

It's not a constraint. inLibrary accepts one or more libraries. In your case, A and X.

[zzvara]

Thanks, I will try that one out.

[zzvara]

@manuzhang That worked. This way I was able to correctly rewrite the classes and references in the corresponding modules. I've added A to the inLibrary. Also I'm providing X V2 to the application on runtime.

[chace86]

@manuzhang That worked. This way I was able to correctly rewrite the classes and references in the corresponding modules. I've added A to the inLibrary. Also I'm providing X V2 to the application on runtime.

How are you providing the X V2 as a runtime dependency? I tried adding % "runtime without success. The dependency is transitive and being evicted.

I am using sbt 1.3.10