datafusion

datafusion copied to clipboard

Support "pre-image" for pruning predicate evaluation

10

## Which issue does this PR close? - Closes #18320. ## Rationale for this change ## What changes are included in this PR? Adding a new rule to expr_simplifier library...

sdf-jkl

Error with union and optimize_projections: Physical input schema should be the same as the one converted from logical input schema

7

### Describe the bug DataFusion fails with schema mismatch error when processing UNION ALL query on parquet files with field metadata. `Error while planning query: Internal error: Physical input schema...

erratic-pattern

Hash partitioning should satisfy hash subset partitioning

2

### Is your feature request related to a problem or challenge? When DF checks if our partitioning is satisfied or if a repartition is needed it only checks if the...

gene-bordegaray

## Which issue does this PR close? - Closes #19272 ## Rationale for this change LogicalPlan::TableScan is currently treated as a leaf node in map_children, but some table providers (such...

shifluxxc

Make `DefaultListFilesCache` work better with prefixed paths

3

### Is your feature request related to a problem or challenge? The current implementation of the `DefaultListFilesCache` stores and retrieves entries from the cache using a provided `Path` as the...

BlakeOrth

Populate partition column statistics for PartitionedFile

2

Superseeds https://github.com/apache/datafusion/pull/15865 Part of https://github.com/apache/datafusion/issues/16800 The idea here was to remove usage of `SchemaAdapter` and at the same time actually populate the partition column statistics.

adriangb

### Describe the bug My understanding, which could be wrong, is that quoted field names should not be treated as placeholder variables when parsing SQL statements. For example: ``` SELECT...

pmallex

Grouped aggregations with many distinct groups do not respect memory limit when input is sorted

2

### Describe the bug In `GroupedHashAggregateStream::spill_previous_if_necessary`, when the `group_ordering` is not `GroupOrdering::None`, spilling is currently not supported. In `GroupedHashAggregateStream::group_aggregate_batch`, there is code that ignores out of memory errors under the...

pepijnve

### Describe the bug I have a query predicate `left = right` where `left` is a column with type `Utf8View` and `right` is a column with type `Dictionary(UInt8, LargeUtf8)`. The...

rroelke

[EPIC] Improved Externalized / Spilling / Large than Memory Hash Aggregation

5

This is a collection of items to improve external (spilling) aggregation ### Background > Abstract—Analytical database systems offer high-performance in-memory aggregation. If there are many unique groups, temporary query intermediates...

alamb