add java 8 to image creation script; remove duplication

[denbkh]

[nchammas]

This is a neat PR.

@shivaram - What is the likelihood that a new set of spark-ec2 AMIs will ever be generated? Do we want to mark files like create_image.sh as "for the record" only, or do we want to actively maintain them?

[shivaram]

I think we will do a round of AMI generation for Spark 2.0 -- this will give us a chance to upgrade to Java 8, Scala 2.11 etc. Does this sound good ?

[douglaz]

Any plans on this?

[shivaram]

Unfortunately I haven't had the time to get around to this yet. I will try to get to it soon or check if there is somebody else interested in helping out with this.

[pcandido]

Hi, I've tested it, but didn't work for me. It seems the 'create_image.sh' file wasn't run on any node. Maybe it needs to be called. I made config manually. I had to:

• install j8
• config alternatives
• set JAVA_HOME in .bash_profile

[douglaz]

@pcandido, this won't run automatically. You manually need to:

1. Launch a machine in Amazon
2. Login, then run the script
3. Create an AMI (Amazon Machine Image) from this instance
4. Then, when you want to launch your cluster, specify this new AMI

Of course, someone can do this process, create a public AMI and make it the default in the spark-ec2 scripts. This is what is really missing here.

[nchammas]

The way we solved this in Flintrock is to dynamically install Java 8 at launch time if it isn't detected on the instance. That avoids the lengthy process of needing to create new AMIs and, at least for Flintrock, it didn't add much time to the launch.

[shivaram]

Thats very useful to know @nchammas - In that case I'd be fine with a change to setup-slave.sh to do this in spark-ec2