flambo
flambo copied to clipboard
Error after submitting job to standalone cluster executor
I've a test project I'm trying to get working, but I can't figure out what's wrong.
After building an uberjar, I submit it to a local standalone Spark cluster:
spark-submit --master spark://txx.attlocal.net:7077 /<path to JAR>/sparktest-0.1.0-SNAPSHOT-standalone.jar
This results in an error on the executor:
org.apache.spark.SparkException: Failed to register classes with Kryo
at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:140)
at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:324)
...
Caused by: java.lang.ClassNotFoundException: flambo.kryo.BaseFlamboRegistrator
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
However, flambo.kryo.BaseFlamboRegistrator
is definitely in the JAR, and the job does execute properly when launched through a REPL:
spark-submit --master spark://txx.attlocal.net:7077 --class clojure.main /<path to JAR>/sparktest-0.1.0-SNAPSHOT-standalone.jar
user=> (require '[sparktest.core])
...
user=> (in-ns 'sparktest.core)
#object[clojure.lang.Namespace 0x11d2dd2d "sparktest.core"]
user=> (-main)
...
100