NoClassDefFoundError when loading assembly jar files with spark package sbt dependencies

[schlichtanders]

trafficstars

Dear ammonite-team,

I just wanted to try out ammonite, loading a custom jar and was suprised that it does not work. I spend hours already and finally have an idea what is going on.

I get a very similar issue like mentioned here https://github.com/lihaoyi/Ammonite/issues/317#issuecomment-322211216 The respective answer mentioned some comments further down issue https://github.com/lihaoyi/Ammonite/issues/317#issuecomment-322517269 doesn't work for my case, unfortunately.

Error printing

In general, there is a problem with the error reporting in ammonite while loading jars.

It looks like everything worked fine, however something behind the scene may have broken. If possible, please enhance your error reporting.

Support spark dependencies in external jars

Unfortunately, this didn't helped me with my original problem. There was indeed a test classpath in the external jar, however deleting it did not lead to success. To reproduce my setting, just build a package with the following sbt (lets name it "my")

name := "my"
version := "0.1.0"
scalaVersion := "2.11.12"
val sparkVersion = "2.1.0"

val sparkDependencies = Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-hive" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-mllib" % sparkVersion
)
libraryDependencies ++= sparkDependencies.map(_ % "provided" withSources())
libraryDependencies += "com.typesafe.scala-logging" %% "scala-logging" % "3.7.2" % "compile" withSources()
libraryDependencies += "com.lihaoyi" %% "sourcecode" % "0.1.4" % "compile" withSources()

as well as the following custom package content

package my
object obj {
  val a = 2
}

and assemble everything together into one big "test.jar" (I build the jar using Intellij Idea, via creating an Artefact, including dependencies and tests, but I would think that using sbt-assembly gives the same results).

The respective test-file I called test.sc contains

// test.sc
println(my.obj.a)

Putting all together, the call to amm results in a NoClassDefFoundError

amm --predef-code 'interp.load.cp(ammonite.ops.Path("/absolute/path/to/my/extended/test.jar"))' test.sc
Compiling test.sc
java.lang.NoClassDefFoundError: my/obj$
  ammonite.$file.test$.<init>(test.sc:1)
  ammonite.$file.test$.<clinit>(test.sc)
java.lang.ClassNotFoundException: my.obj$
  java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  ammonite.runtime.SpecialClassLoader.findClass(ClassLoaders.scala:198)
  java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ammonite.$file.test$.<init>(test.sc:1)
  ammonite.$file.test$.<clinit>(test.sc)

any help is highly appreciated!

versions I am using:

• OS: centos 7 (in virtualbox on Windows 10)
• ammonite: 1.0.3
• scala: 2.11.12

[alexarchambault]

It could work with ammonite-spark, although I haven't tried to run Ammonite scripts with it.

[erikerlandson]

I've been trying to do interp.load.cp in the latest almond (0.1.7) and nothing works

[erikerlandson]

I put this here: https://github.com/jupyter-scala/jupyter-scala/issues/173, which seems more appropriate.