Azure
GPU images set work dir to $SPARK_HOME but base images do not
https://github.com/Azure/aztk/blob/f7c1cb51729ce5347ebd7a732d5b735142c1332c/docker-image/gpu/spark1.6.3/Dockerfile#L78
This should be set on base, so it is propagated down to all other images.
@jiata What do you think about this. On second thought, it might be better to have the working dir just be /. What do you think?
I'm not sure I like having this at the root since it could potentially just spew a bunch of stuff out there. Having it in a known location seems better to me.
Makes sense. I am just not sure that $SPARK_HOME is that place.
Maybe /root is best then.
Based on the definition of WORKDIR(https://docs.docker.com/engine/reference/builder/#workdir) it will only apply to the very last command in the container right? Since the last command is running Spark I think $SPARK_HOME does seem like the right place to me.
WORKDIR is also the location that will be defaulted to on a /bin/bash into the container (for us that means the default location for SSHing in).
Also that last line in the Dockerfile is actually not used. We specify a command in docker run which overrides that line.
So the reason I don't think WORKDIR should be $SPARK_HOME is that this is the default location in the container. It would be expected to be the home directory of the user, which for root is /root. We could also set $HOME to this since I believe that defaults to /.
working dir set to / feels the most intuitive imo - when you ssh in, and you appear in $SPARK_HOME, users may be confused...
