NameNode is not starting

[nareshbab]

Hi, Thanks for the awesome script for setting up spark on gce. I am able to setup cluster on gce but my namenode won't start. Logs: i@hdp99-spark-cdh-master:/opt/ephemeral-hdfs/sbin$ ./start-dfs.sh Starting namenodes on [hdp99-spark-cdh-master] Error: Please specify one of --hosts or --hostnames options and not both. hdp99-spark-cdh-slave0: datanode running as process 27587. Stop it first. Starting secondary namenodes [0.0.0.0] Error: Please specify one of --hosts or --hostnames options and not both. i@hdp99-spark-cdh-master:/opt/ephemeral-hdfs/sbin$ ./stop-dfs.sh Stopping namenodes on [hdp99-spark-cdh-master] Error: Please specify one of --hosts or --hostnames options and not both. hdp99-spark-cdh-slave0: stopping datanode Stopping secondary namenodes [0.0.0.0] Error: Please specify one of --hosts or --hostnames options and not both.

It would be really helpful, if you guys can help me.

Thanks in Advance !!!

[broxtronix]

Hello @nareshbab! Thanks for trying the script out!

If I'm not mistaken, I believe that hdfs should already be running when you log into your cluster... the configure_and_start_hadoop() function in spark_gce.py launches a hadoop namenode on the master and configures the slaves, and it calls the start_dfs.sh script. Although now that I stop and start HDFS via stop-dfs.sh and start-dfs.sh, I see the same error you are reporting. Out of curiosity, can you run commands such as hdfs dfs -ls <HDFS path>?

I'll confess that although I use Spark extensively on GCE using this script, I have hardly used HDFS so it may not be set up correctly. Unfortunately I'm not familiar with HDFS configuration, so we may need to work together to find the solution here. When I implemented the script, I pulled over the configuration for HDFS that was carried out by the official spark-ec2 script bundled with Spark and confirmed that Hadoop was running, but I have not used HDFS extensively from outside or inside of Spark. If you have more familiarity with HDFS and are able to poke around a bit and suggest configuration tweaks, that would be a tremendous help!

[nareshbab]

Hi @broxtronix I tried the hdfs dfs -ls and the issue I am getting is ls: Call From hdp99-spark-cdh-master.c.<Project ID>.internal/10.240.9.169 to hdp99-spark-cdh-master:9000 failed on connection exception: java.net.ConnectException: Connection refused; I agree for the fact that configure_and_start_hadoop() installs and runs the hadoop cluster, but for some reason namenode won't start. All the datanodes are running and also get listed by jps. I am working on fixing the issue, will update you soon.