ammonite-spark icon indicating copy to clipboard operation
ammonite-spark copied to clipboard

Warn if suspicious conf is found

Open alexarchambault opened this issue 5 years ago • 1 comments

Typically, if spark.yarn.archive or spark.home in the spark conf, or SPARK_HOME in the env, are set, these may clash with the way ammonite-spark handles things (like the spark.yarn.archive would make executors have an unexpected classpath, …). A warning message should be printed in that case.

alexarchambault avatar Aug 22 '18 15:08 alexarchambault

Will these clash or will they potentially be compatible as suggested by @dyno in https://github.com/dyno/ammonite_with_spark_on_emr/blob/master/Makefile#L1.

If it is potentially compatible, could you explain a bit more about how $SPARK_HOME works and how it might conflict with the way ammonite-spark handles things? This will be important for generalizing support for provided spark distribution outside of EMR clusters.

mpacer avatar Oct 29 '18 23:10 mpacer