spark-ec2
spark-ec2 copied to clipboard
amplab
Scripts used to setup a Spark cluster on EC2
Some conf change requires cluster restartup to take effect, say 'SPARK_WORKER_OPTS'; but during stop/start spark cluster via spark-ec2, it seems it re-setup cluster and flushes all of conf. So is...
In the documentation there is a broken link reference: "If you wish to start a cluster using Spark, please refer to http://spark-project.org/docs/latest/ec2-scripts.html"
This would be very, very nice in 2016/2017. Or at least provide some instructions in README on how to do so.
With Spark cluster launched and started with ec2 script, is there the Hadoop cluster ready to go, or just an easy command to start it? The reason I am asking...
Scripts work great, but EC2 machines have Java 1.7 on them rather than Java 1.8. I can see https://github.com/amplab/spark-ec2/pull/12 adds some functionality related to this, but it's not clear to...
Hello all, Could you please test the changes for SPARK_TACHYON_MAP for spark 2.0.0 and 2.0.1 Many thanks! Sanjay
Submitting a job: `/asidev/spark-latest/bin/spark-submit --class com.cxxxxx.ABC --master spark://ip-10-100-111-111.ec2.internal:7077` Has the following exception. Those servers are in private VPC. Running Spark using the REST application submission protocol. Using Spark's default log4j...
Right now it seems the latest hadoop version in spark-ec2 is 2.4, but actually in spark download page, it can be up to 2.7, also it is available in aws...