Jacob Freck

GPU images set work dir to $SPARK_HOME but base images do not

6

https://github.com/Azure/aztk/blob/f7c1cb51729ce5347ebd7a732d5b735142c1332c/docker-image/gpu/spark1.6.3/Dockerfile#L78 This should be set on base, so it is propagated down to all other images.

add support for arbitrary submit flags

2

users should be allowed to specify flags and arguments when submitting apps. `aztk spark cluster submit --id --name /path/to/program.py -flag1 arg1 -flag2 arg2` In the above example, `-flag1 arg1 -flag2...

Nvidia docker 2.0 just released. https://github.com/nvidia/nvidia-docker/wiki/Installation-(version-2.0)

Described here: https://spark.apache.org/docs/latest/security.html We should set the spark.authenticate.secret parameter by default using a shared secret.

Currently, all files are uploaded to the tasks working directory on the node. This causes issues if a user wants to reference their files as they do not know where...

Add support for Flink https://flink.apache.org/