spark-ec2 script failed to "mkdir -p ~/.ssh"

[deanwampler]

I don't know why this didn't work on my MacBook Pro, but I suspect I was prompted to allow the ip address to be added to my known-hosts file, but the prompt was eaten by the python script. Turning off my firewall didn't help.

Anyway, fortunately, the failed command to ssh to the master and run this mkdir command was echoed to my terminal, so I could copy and run it myself, then restart the launch script replacing "--copy" with "--resume". That appeared to work (mostly; I discovered later that the wikipedia data wasn't in HDFS...).

You might add this to the list of "FAQs" you already have on the tutorial's setup page.

[UFMurphy]

Hahaha. Of course it has to be the infamous Dean Wampler to figure this out. This was annoying the hell out of me. Thanks, Dean. I hope all is well.

[deanwampler]

You said "infamous", but I think you meant to say "miscreant"...

;)

[rayshih]

I face this problem just 30 mins ago. I figure it out that this is just a problem that the bootstrap of the instance is too slow, such that, the ssh is not ready while this script trying to send this command.

Here is how I solve this problem: add -w 300 to make the waiting time longer.