Loading jars using lib folder

[rinnyB]

Hey guys, I've got Scala [2.11.8] Spark [2.1.0] Hadoop [2.8.0] {Hive ✓} on Ubuntu 16.04. It is installed from zip archive. I'm running it from spark-notebook folder using ./bin/spark-notebook I'm trying to add a simple jar through lib folder under spark-notebook. And, the problem is it doesn't work.

Here's my source for jar:

package sample
object people {
  class Person (age: Int, name: String) {
    def say: Unit = {
	println ("Hi, I'm" + name)
    }  
  } 
}

sbt file:

name := "sample"
version := "1.0"
organization := "org.example"
scalaVersion := "2.11.8"

Here's a result of import from jar located under spark-notebook/lib folder using sbt console

And here's spark-notebook one

Interestingly it works for spark-notebook 0.6.3 and spark 1.6.3 version under Windows 10 with same notebook setup. (0.7.0+ wont work on windows :( )

Hope for your help. Thanks in advance!

[mambusskruj]

+1

[andypetrella]

Indeed, the adding the jar into the lib folder isn't enough, this is because the spark-notebook script/s are listing explicitly all jars added to the classpath.

Also, another problem will happen if you want to use these jars on the executors because they won't be added to the Spark configuration.

The recommended way would be (let me know if make sense, and worth improving):

1. Add the path in the var env EXTRA_CLASSPATH: it uses : as paths separator and will add the jar into the driver's classpath
2. Add the path in the var env ADD_JARS: this one uses , as paths separator and will the jar into the executors' classpath

So you could do something like:

# copy the jar
cp my-lib.jar lib

# launch with my-lib in both driver and executors classpaths
EXTRA_CLASSPATH=../lib/my-lib.jar ADD_JARS=../lib/my-lib.jar bin/spark-notebook

HTH