EC2 setup does not work for any user but root

[kuribe]

Hi,

I was trying to use the spark-ec2 script from Spark to create a new Spark cluster wit an user different than root (--user=ec2-user). Unfortunately the part of the script that attempts to copy the templates into the target machines fail because it tries to rsync /etc/* and /root/*

This is the full traceback

rsync: recv_generator: mkdir "/root/spark-ec2" failed: Permission denied (13) *** Skipping any contents from this failed directory ***

sent 95 bytes received 17 bytes 224.00 bytes/sec total size is 1444 speedup is 12.89 rsync error: some files/attrs were not transferred (see previous errors) (code 2 3) at main.c(1039) [sender=3.0.6] Traceback (most recent call last): File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1455, in main() File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1447, in main real_main() File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1283, in real_main setup_cluster(conn, master_nodes, slave_nodes, opts, True) File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 785, in setup_cluster modules=modules File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1049, in deploy_files subprocess.check_call(command) File "/usr/lib64/python2.7/subprocess.py", line 540, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['rsync', '-rv', '-e', 'ssh -o StrictHos tKeyChecking=no -o UserKnownHostsFile=/dev/null -i /home/ec2-user/.ssh/sparkclus terkey_us_east.pem', '/tmp/tmpT4Iw54/', u'[email protected] zonaws.com:/']' returned non-zero exit status 23

Is there a workaround for this? I want to improve security of our operations by avoiding user root on the instances.

[shivaram]

Unfortunately the spark-ec2 scripts are only designed to work as the root user right now. There isn't any simple workaround and I think supporting non-root users will require extensive code changes.

[adamjk]

Should probably remove that --user parameter then.

[nchammas]

@adamjk - Good call. Is that option useful for anything? I've never used it. Perhaps we should remove it?

[shivaram]

+1 on removing the option.

[agustinbf]

I agree. Please remove the option as is confusing.