Andrew Lamb

[EPIC] Streaming partitioned writes

11

### Is your feature request related to a problem or challenge? This is a tracking epic for a collection of features related to writing data. The basic idea is better...

Add support for `CopyTo::partition_by` field in datafusion_proto

4

After https://github.com/apache/arrow-datafusion/pull/9240 datafusion protobuf will will silently ignore partition_by options; The task is 1. add code to serialize / deserialize the `CopyTo::partition_by` field, added in https://github.com/apache/arrow-datafusion/pull/9240 to the existing serialization...

Panic when reading from partitioned datasets with columns that have `'` in them

1

### Describe the bug There is a bug when reading from partitioned tables that have commas in their names Here is the test https://github.com/apache/arrow-datafusion/blob/b2a04519da97c2ff81789ef41dd652870794a73a/datafusion/sqllogictest/test_files/copy.slt#L109 ### To Reproduce Run this script...

Draft while I figure out: - [ ] Get tests passing - [ ] Review string rewrite / concat in the rewriter and figure out how/why it is necessary (doesn't...

Support writing hive style partitioned files in `DataFrame::write` command

3

### Is your feature request related to a problem or challenge? @Omega359 asked on discord: https://discord.com/channels/885562378132000778/1166447479609376850/1207458257874984970 Q: Is there a way to write out a dataframe to parquet with hive-style...

Finalize SIGMOD 2024 paper ~(if accepted)~

23

### Is your feature request related to a problem or challenge? @JayjeetAtGithub @Dandandan @yjshen @ozankabak @sunchao and @viirya and submitted a paper to the [SIGMOD 2024 conference](https://2024.sigmod.org), which was tracked...

Support `IS NOT NULL` predicates in PruningPredicate

3

### Is your feature request related to a problem or challenge? the [`PruningPredicate` ](https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.PruningPredicate.html)currently does not handle `IS NOT NULL` predicates ### Describe the solution you'd like _No response_ ###...

Support ORDER BY in AggregateUDF

6

### Is your feature request related to a problem or challenge? Some built in aggregates (such as `FIRST_VALUE`, `LAST_VALUE` and `ARRAY_AGG`) support an optional ORDER BY argument that defines the...

Invalid timezone information is ignored after `Z` in timestamp literals

7

### Describe the bug Some timestamps that should error are actally parsed, leading to confusing cases when users make a mistake For example this timestamp is invalid `'2023-12-05T21:58:10.45ZZTOP'` but it...

Implement spilling for PartialSortExec

2

### Is your feature request related to a problem or challenge? `PartialSortExec` was added in https://github.com/apache/arrow-datafusion/issues/7456 / https://github.com/apache/arrow-datafusion/pull/9125 While one of the major benefits of this operator is to reduce...