Andy Grove

Add option to replace SortMergeJoin with ShuffleHashJoin

1

### What is the problem the feature request solves? Other Spark accelerators, such as Spark RAPIDS and Apache Gluten, replace SortMergeJoin with ShuffleHashJoin for improved performance. We should evaluate this...

chore: Remove NativeBase static initializer (to improve error handling when native lib fails to load)

1

## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/999 ## Rationale for this change If the static initialization block failed to load the library then we did not see the...

### What is the problem the feature request solves? This issue is for tracking issues to perform Comet's performance when running benchmarks derived from TPC-DS. - [ ] Add support...

feat: Implement shared memory pool for case where spark.memory.offHeap.enabled=false

2

## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/996 ## Rationale for this change Simplify memory configuration. ## What changes are included in this PR? Allocate one shared pool per...

Plugin can fail to initialize native library and hide the root cause

3

### Describe the bug I tried running Comet in k8s, and got an error initializing NativeBase, but no root cause was given. ``` │ 24/10/07 22:42:11 WARN CometSparkSessionExtensions: Comet extension...

perf: Apply DataFusion's projection pushdown rule

3

## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/908 ## Rationale for this change Improve performance (and reduce memory overhead) of `HashJoinExec` by pushing down projection into the join leveraging...

### What is the problem the feature request solves? As described in detail in https://github.com/apache/datafusion-comet/issues/949, when using native memory management, we currently create one pool per plan and do not...

Support RangePartitioning with native shuffle

7

### What is the problem the feature request solves? We do not currently support RangePartitioning with native shuffle. Adding this support would allow us to use native shuffle for more...

Add support for bloom_filter_agg

6

### What is the problem the feature request solves? Some TPC-H queries use `bloom_filter_agg`, and Comet does not have a native implementation yet. A workaround is to set `spark.sql.optimizer.runtime.bloomFilter.enabled=false`. ###...

SumDecimal always returns nullable=true

1

### Describe the bug SumDecimal currently hardcodes nullable=true, and this is correct when ANSI mode is not enabled, because overflows cause null values. However, in ANSI mode, overflows cause exceptions....