ammonite-spark
ammonite-spark copied to clipboard
Warn if suspicious conf is found
Typically, if spark.yarn.archive
or spark.home
in the spark conf, or SPARK_HOME
in the env, are set, these may clash with the way ammonite-spark handles things (like the spark.yarn.archive
would make executors have an unexpected classpath, …). A warning message should be printed in that case.
Will these clash or will they potentially be compatible as suggested by @dyno in https://github.com/dyno/ammonite_with_spark_on_emr/blob/master/Makefile#L1.
If it is potentially compatible, could you explain a bit more about how $SPARK_HOME works and how it might conflict with the way ammonite-spark handles things? This will be important for generalizing support for provided spark distribution outside of EMR clusters.