spark-ec2
spark-ec2 copied to clipboard
amplab
Scripts used to setup a Spark cluster on EC2
I faced following issue while running `./spark-ec2 --key-pair= --identity-file= --region=us-west --instance-type=t2.micro -s 2 launch test-cluster`: ``` [...] Initializing spark --2016-07-28 03:58:47-- http://s3.amazonaws.com/spark-related-packages/spark-1.6.2-bin-hadoop1.tgz Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.40.74 Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.40.74|:80......
When creating cluster using --hadoop-major-version yarn I noticed the hdfs does not function. I relaunched the cluster using --hadoop-major-version 2 and hdfs works fine. root@... $ /root/ephemeral-hdfs/bin/hadoop fs -cp "s3n://[redacted]"...
## Description `--spark-ec2-compressed` option enables to precise a compressed version of spark-ec2. This option is an alternative to cloning spark-ec2 from GitHub. ## Accepted compression format .tar, .tar.gz, .tar.bz2, .tar.xz
# Configuration file `-c` or `--conf-file` option enables to use a YAML configuration file to simplify the way you interact with the cluster. ## Launch new cluster `./spark-ec2 --conf-file config.yml...
# Elastic Spark Cluster ## Prerequisite A running Spark cluster. ## add-slaves `./spark-ec2 -k key-name -i file.pem add-slaves cluster_name` This command adds a new slave to the cluster. To add...
s3a is the successor of s3n file system, s3a offers higher performance and support of larger files. For more details: [https://wiki.apache.org/hadoop/AmazonS3](https://wiki.apache.org/hadoop/AmazonS3)
See https://docs.python.org/2.7/library/argparse.html#upgrading-optparse-code for a checklist of things that should have been done (which I tried to follow), and http://stackoverflow.com/questions/3217673/why-use-argparse-rather-than-optparse for a discussion of why this is a good idea in...
There is no ami-list entry for eu-central-1 (Frankfurt)
Hey everybody, due to company policy restrictions, we are just allowed to use the eu-central-1 AWS region -- unfortunately, the Spark-EC2 image is not added there... Would be really great...
sparc
I am using spark-1.6.1-prebuilt-with-hadoop-2.6 on mac. I am using the spark-ec2 script to launch a cluster in Amazon VPC. The setup.sh script [run first thing on master after launch] uses...
