Andy Grove

Ballista: Fix hacks around concurrency=2 to force hash-partitioned joins

2

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** By default, DataFusion uses hash-partitioned joins if concurrency > 1 which led...

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** There is currently no way to specify configuration parameters such as default...

Ballista: Executor must return statistics in CompletedTask / CompletedJob

2

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** We cannot fix the shuffle mechanism until we have partition stats, or...

**Describe the bug** I can't run the integration tests on an M1 MacBook Air. ``` apache/arrow-datafusion#6 2.909 musl-gcc -I. -Iinclude -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -fPIC -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC...

Ballista: UnresolvedShuffleExec and ShuffleReaderExec should show correct partitioning scheme

1

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Once https://github.com/apache/arrow-datafusion/pull/750 is merged, `UnresolvedShuffleExec` and `ShuffleReaderExec` work correctly but they both...

Ballista should serialize Parquet statistics

1

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** When the Ballista scheduler or executor deserializes a ParquetExec it collects the...

Ballista assumes all aggregate expressions are not DISTINCT

5

**Describe the bug** We have a hard-coded `distinct = false` parameter in `ballista/rust/core/src/serde/physical_plan/mod.rs`. ``` rust Ok(create_aggregate_expr( &aggr_function.into(), false, //

[DF] Add optimizer rules to translate subqueries to joins

1

Add optimizer rules to translate subqueries to joins (when possible)

[DF] Implement subquery decorrelation optimizer rules

1

**Is your feature request related to a problem? Please describe.** PR https://github.com/apache/arrow-datafusion/pull/2885 adds three new optimizer rules for decorrelating subqueries and translating them into joins. This may result in more...

[DF] DataFusion Missing Features & Bugs

2

The purpose of this issue is to track the work that we need to do in the [DataFusion](https://github.com/apache/arrow-datafusion) project to support moving the dask-sql planner to DataFusion. ## High Priority...