docker-spark-cluster icon indicating copy to clipboard operation
docker-spark-cluster copied to clipboard

Published 20 hours ago • verified publisher icon mvillarrealb

Refactoring infrastructure

Open kkeker opened this issue 5 years ago • 2 comments

  • Change base image (Oracle Entreprise Linux 8).
  • Improved the use of variables.
  • Redid the installation of python packages on pip.
  • Redid installing jvm components on sdkman.

kkeker avatar Jan 30 '20 06:01 kkeker

I like how $SPARK_HOME is used here. It makes things more clear for me.

EmileSonneveld avatar Dec 13 '22 14:12 EmileSonneveld

I like how $SPARK_HOME is used here. It makes things more clear for me.

The main application in these containers is Spark, so it's important to know where it's installed. If you need to enter a container, the SPARK_HOME directory will be used as the base directory when you enter, thanks to WORKDIR ${SPARK_HOME}, this will make it easy to see the installed components simply by using ls or running commands from the relative path ./. Also, the main process startup scripts use this variable instead of the hardcoded absolute paths.

kkeker avatar Dec 13 '22 14:12 kkeker