spark-sql-perf icon indicating copy to clipboard operation
spark-sql-perf copied to clipboard

point at spark master?

Open dotnwat opened this issue 8 years ago • 5 comments

This is probably a really simple question. How can I point bin/run to a running Spark cluster?

dotnwat avatar Feb 29 '16 23:02 dotnwat

I also has the same question. It seems that this can only run on localhost

GuanhuaWang avatar Mar 11 '16 01:03 GuanhuaWang

Can someone provide tutorial to run on spark? And how to import TPC-DS.

gaohannk avatar Mar 11 '16 04:03 gaohannk

Yeah, definitely

GuanhuaWang avatar Mar 11 '16 04:03 GuanhuaWang

Go to this directory -> spark-sql-perf-master/src/main/scala/com/databricks/spark/sql/perf and in file RunBenchmark.scala change .setMaster("local[*]") to pint to your master. Hopefully this should work

hchawla1 avatar Mar 27 '16 18:03 hchawla1

Local: val conf = new SparkConf().setAppName("appname").setMaster("local[*]"); val sc = new SparkContext(conf); Cluster mode:

Start spark master if required worker also (eg: start-all.sh (SPARK_HOME/sbin))

By default u can get spark master url by giving http://localhost:8080 then you can see the spark master url.

val conf = new SparkConf().setAppName("appname").setMaster("spark://localhost:7077"); val sc = new SparkContext(conf);

i think this will help you run your example

madhugithub2014 avatar Aug 03 '16 10:08 madhugithub2014