Andy Grove

Possible native shuffle optimization

1

### What is the problem the feature request solves? I noticed that we execute each query stage with two separate native plans. For example, here is the first query stage...

Implement Common Subexpression Elimination optimizer rule

8

### What is the problem the feature request solves? When running TPC-H q1 in Spark/Comet, the expression `l_extendedprice#21 * (1 - l_discount#22)` appears twice in the query and currently gets...

### What is the problem the feature request solves? We currently unpack dictionaries before SortExec, which seems inefficient. I experimented with removing this unpacking and was able to run TPC-H...

Improve performance of Spark-compatible decimal aggregates

1

### What is the problem the feature request solves? The benchmarks added in https://github.com/apache/datafusion-comet/pull/948 show that Comet's Spark-compatible aggregates are ~50% slower than the DataFusion equivalents: ``` aggregate/avg_decimal_datafusion time: [653.56...

Performance regression after adding support for SMJ with join filter

9

### Describe the bug When running TPC-DS benchmarks against 100 GB data set I see a large regression in performance. For example, here are the timings for q72 before and...

Fall back to Spark if query uses DPP to avoid perf regressions in TPC-DS

1

### What is the problem the feature request solves? Comet does not yet support DPP and this can result in poor performance on the TPC-DS benchmark due to scanning more...

Optimize filters to remove redundant IsNotNull checks

4

### What is the problem the feature request solves? I am comparing native query plans between Comet and Ballista for TPC-H q1 and noticed a significant difference between the filter...

CometSparkToColumnar should have different name for row vs columnar input

3

### What is the problem the feature request solves? I would like to change the `nodeName` for `CometSparkToColumnar` as follows: ```scala override def nodeName: String = if (child.supportsColumnar) { "CometSparkColumnarToColumnar"...

Implement native parsing of CSV files

1

### What is the problem the feature request solves? We can probably accelerate reading of CSV files by continuing to use JVM Spark to read bytes from disk but then...

### What is the problem the feature request solves? In the native planner, we insert a CopyExec node into the plan. The timings from this operator are not visible in...