flambo icon indicating copy to clipboard operation
flambo copied to clipboard

Error after submitting job to standalone cluster executor

Open xiongtx opened this issue 6 years ago • 0 comments

I've a test project I'm trying to get working, but I can't figure out what's wrong.

After building an uberjar, I submit it to a local standalone Spark cluster:

spark-submit --master spark://txx.attlocal.net:7077 /<path to JAR>/sparktest-0.1.0-SNAPSHOT-standalone.jar

This results in an error on the executor:

org.apache.spark.SparkException: Failed to register classes with Kryo
	at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:140)
	at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:324)
...
Caused by: java.lang.ClassNotFoundException: flambo.kryo.BaseFlamboRegistrator
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

However, flambo.kryo.BaseFlamboRegistrator is definitely in the JAR, and the job does execute properly when launched through a REPL:

spark-submit --master spark://txx.attlocal.net:7077 --class clojure.main /<path to JAR>/sparktest-0.1.0-SNAPSHOT-standalone.jar

user=> (require '[sparktest.core])
...

user=> (in-ns 'sparktest.core)
#object[clojure.lang.Namespace 0x11d2dd2d "sparktest.core"]

user=> (-main)
...
100

xiongtx avatar Dec 16 '18 04:12 xiongtx