datafusion

datafusion copied to clipboard

DataFusion weekly project plan (Andrew Lamb) - April 29, 2024

3

Follow on to https://github.com/apache/datafusion/issues/10172 **It would be great for other contributors to DataFusion who plan non trivial work could try to make them visible somehow as well** 🙏 (feel free...

alamb

Support simplification that requires multiple applications of constant folding / simplification

7

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** The constant folding and constant evaluation in [constant_folding.rs](https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/optimizer/constant_folding.rs)(and added to in https://github.com/apache/arrow-datafusion/pull/1153)...

alamb

Make all SchemaProvider trait APIs async

4

### Is your feature request related to a problem or challenge? Currently, only `table` API is async, but `register_table`, `deregister_table`, `table_names`, and `table_exist` can be also async. Also, most of...

metesynnada

Document Sort Merge Join algorithm

4

### Is your feature request related to a problem or challenge? I faced that it is challenging to understand fully how Sort Merge Join `sort_merge_join.rs` implemented and would be nice...

comphead

Optimized version of `SortPreservingMerge` that doesn't actually compare sort keys of the key ranges are ordered

15

### Is your feature request related to a problem or challenge? When merging a large number of pre-sorted streams (e.g. in our case, a large number of pre-sorted parquet files)...

alamb

Remove ScalarFunctionDefinition

5

## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/10175. ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested? ## Are there...

lewiszlw

Any plan to implement datafusion go/c++ binding?

6

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** A clear and concise description of what the problem is. Ex. I'm...

xiaofan-luan

~Draft as it builds on https://github.com/apache/datafusion/pull/10331~ ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/10181 ## Rationale for this change Rationale: There important use cases for creating `Expr`s and executing...

alamb

Support limit in StreamingTableExec

9

## Which issue does this PR close? Closes todo. ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested? ## Are there...

lewiszlw

Schema incorrect after select over aggregate function that returns a different type than the input

2

### Describe the bug When you have a column that is created using `array_agg` and the column name is the same as the column name of the original data, calling...

timsaucer