spark-ec2 icon indicating copy to clipboard operation
spark-ec2 copied to clipboard
verified publisher icon amplab

How can launch Spark 2.0.1 and Hadoop 2.7?

Open biolearning opened this issue 9 years ago • 2 comments

Right now it seems the latest hadoop version in spark-ec2 is 2.4, but actually in spark download page, it can be up to 2.7, also it is available in aws s3 http://s3.amazonaws.com/spark-related-packages. So the question is - how to launch such Spark cluster and if it does not support is there any workaround?

--hadoop-major-version=HADOOP_MAJOR_VERSION Major version of Hadoop. Valid options are 1 (Hadoop 1.0.4), 2 (CDH 4.2.0), yarn (Hadoop 2.4.0) (default: yarn)

biolearning avatar Oct 25 '16 05:10 biolearning

See #56

shivaram avatar Oct 25 '16 17:10 shivaram

Thanks. When is that improvement expected to check into the 2.0 branch?

biolearning avatar Oct 27 '16 07:10 biolearning