Daft icon indicating copy to clipboard operation
Daft copied to clipboard

refactor: convert scan task `split_by_row_groups` and `merge_by_sizes` into optimizer rule

Open universalmind303 opened this issue 8 months ago • 3 comments

Is your feature request related to a problem? Please describe.

Currently there is an optimization inlined inside translate_single_logical_node.

I think it'd make the code a bit easier to reason about if this was instead extracted into an optimizer pass.

Describe the solution you'd like split_by_row_groups and merge_by_sizes is moved out of translate_single_logical_node and instead moved into a dedicated optimizer rule

Describe alternatives you've considered Keep the code as is. It technically works, but it does make the code a bit more difficult to reason about.

universalmind303 avatar Jun 20 '24 20:06 universalmind303