spark-sql-perf

spark-sql-perf copied to clipboard

I'm running spark-sql-perf in a scenario where you would cache tables in RAM for performance reasons. Code here. The result is blockmanager failing, see the trace below... val sqlContext =...

slavitch

dataGen fails

1

Spark 2.0.1, 6/11/02 15:28:45 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, 172.16.90.5): java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: 0 if (assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object).isNullAt) null...

slavitch

I am new to spark-sql-perf test tool. I have some of confused questions about it. Hopefully got your answers. Thanks in advance ! 1) I deployed Spark 1.6.2 cluster. Could...

jameszhouyi

Hi experts, I am using: spark 1.6.1 scala 2.10.5 I get many errors when run following command: ./bin/run -benchmark DatasetPerformance The error is: [info] Compiling 40 Scala sources to /data/tpc-ds/spark-sql-perf-master/target/scala-2.10/classes......

BellCui

Improvement ideas - more options

2

Hi, I'd like to contribute the below changes, looking for second opinions on whether this is the direction we want to go with this benchmark. I suggest the following options...

a-roberts

point at spark master?

5

This is probably a really simple question. How can I point bin/run to a running Spark cluster?

dotnwat

TPC-DS.. dataGen.. format

2

table.genData(tableLocation, **format**, overwrite, clusterByPartitionColumns, What value does format take when generating TPC-DS benchmarks?

kgebaly

Looking at [the most recent Travis build](https://travis-ci.org/databricks/spark-sql-perf/builds/119616966) we see: ``` 16/03/30 19:02:44 INFO SessionState: Created HDFS directory: file:/tmp/scratch-362b2e98-2401-4066-9a73-06f6dbbc3454/travis/447f0c84-1866-45d9-aeca-53fb780ef096/_tmp_space.db Internal error when running tests: java.lang.OutOfMemoryError: PermGen space Exception in thread "Thread-1"...

jpallas

Which TPC-DS queries are supposed to be working? I'm finding that several queries are not parseable, for a variety of reasons. Specifically, the following queries compile without exception: Set(q25, q65,...

JonathanMace

Is there a sample script to run the benchmarks?

2

Is there a simple starter script to begin using the repository. I am new to scala, so any guidance will be really helpful.

krdeepakio