Got an issue with trying the spark-shell wordcount example.

[ajrader]

when I tried to launch the spark shell example in https://github.com/alexholmes/vagrant-hadoop-spark-hive/blob/master/VALIDATING.md I was not able to get a live spark context (sc).

This was because the sparkContext had already closed down. I found that if i launched it using

spark-shell

instead of

spark-shell --master yarn-client

I was able to run through this example.

Basically I let spark manage the VM memory rather than try to specify it. see this stackoverflow thread: http://stackoverflow.com/questions/27792839/spark-fail-when-running-pi-py-example-with-yarn-client-mode

[joaonunesk]

If you don't specify the --master yarn-client arguments, then the Spark application won't run in the YARN cluster.

For the sake of testing, you can solve it by adding the following properties to the YARN configurations options file (i.e. /usr/local/hadoop-2.7.2/etc/hadoop/yarn-site.xml) and by restarting the Hadoop services.

<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

More info at https://stackoverflow.com/a/43302946