spark-scala-playground

spark-scala-playground copied to clipboard

How join reorder works.

5

Hi @bartosz25 , Spark provides `org.apache.spark.sql.catalyst.optimizer.ReorderJoin` for more efficient join. I notice that you have written 2 test cases about ReorderJoin in your article: https://www.waitingforcode.com/apache-spark-sql/spark-sql-operator-optimizations-part-2/read But, I think these two...

bithw1

whole stage generated code for a simple query

6

Hi, @bartosz25 , I have a simple test case that would like to see whole stage generated code ``` test("whole stage code gen test") { val spark = SparkSession.builder().enableHiveSupport().master("local").appName("whole stage...

bithw1

About adaptive execution

2

Hi, @bartosz25 In the coming Spark 3.0 release or in the master branch of Apache Spark git, The feature `adaptive execution` is mature, it will be released in Spark 3.0...

bithw1

Why spark generate Java code and not scala code?

6

igreenfield