Give example of a small Spark cluster

[muelli]

It'd be nice to see how to use the role to create a small Spark cluster.

[helt]

I am working on this one (at least, i want to share it, once i got it up and running). However, I am uncertain if configuration is correct and how to proceed to start the spark cluster... The spark-master is running using the standalone cluster config (by manually triggering sbin/start-master.sh), but I am not successful to start slaves. They dont start and therefore not connect to the master.

So the question I have for proceeding is: How does azavea start the spark cluster (e.g. which mode; standalone, mesos, or yarn?), and what are the requirements for it? (Do you have a more elaborate example usage somewhere on github?) Is it sufficient to ansible the azavea/ansible-hdfs and the azavea/ansible-spark ? To me, it appears like something basic does not work correctly in my setup

[helt]

ok. just a hint (for me):

When running in the spark cluster in standalone mode, then the spark_user_shell should be /bin/bash, not /bin/false

[lhoss]

This is coming very soon , thx to molecule tests , see https://github.com/azavea/ansible-spark/issues/28 👍