spark-sql-perf
spark-sql-perf copied to clipboard
databricks
Before this change, logic in ExperimentStatus would cause query planning to be prewarmed as a side effect of capturing the current plan. This change removes this side effect, but adds...
I am trying to generate data for TPC-DS and data is getting generated but while analyzing the table it throws exception. This is the log: It starts analyzing the tables...
Build errors. spark 3.0.0 build.sbt snippet ... crossScalaVersions := Seq("2.11.12", "2.12.10") ... sparkVersion := "3.0.0" [info] Compiling 66 Scala sources to /opt/spark-sql-perf/target/scala-2.12/classes... [info] 'compiler-interface' not yet compiled for Scala 2.12.10....
Noticed that spark 3.0.0 does not have method org.apache.spark.sql.SQLContext.createExternalTable, updated the TPC-DS setup part to support spark 3.0.0 by using SparkSession to create table. Thanks for your reviewing.
We are looking to validate if the returned results of TPC-DS queries are correct. We used IBM's TPC-DS benchmark suite before, which has a process of validating the correctness of...
This PR fixes table creation in case of csv format data. 1. It add header while generating data. 2. It infers schema and header while creating external tables.
I splited the Generate data and Query query in two jar files. Firstly generated data and then parallelly query the data. Most of tasks cost about 20ms. However some tasks...
Is it okay to configure "executor-memory=1G" ? I am running spark querying tpcds.tpcds2_4Queries(q1-q99) testing on 100G data on kubernetes cluster. I want to find out the most suitable executor-memory for...
According to the README, the data is on local disk; is there anyway to put into some external storage, e.g. object storage?
I'm confused about what is the difference between tpcds2_4Queries and tpcds1_4Queries? my spark version is 2.3.2 and scala version is 2.11.8.
