datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Stop copying LogicalPlan and Exprs in `PushDownFilter`

Open alamb opened this issue 1 year ago • 1 comments

Is your feature request related to a problem or challenge?

Part of https://github.com/apache/datafusion/issues/9637

As part of making the planner faster, we are updating the optimizer passes to avoid copying LogicalPlan and Expr (see https://github.com/apache/datafusion/issues/9637)

Describe the solution you'd like

I would like to reduce the amount of copying in this pass (even though it doesn't appear in current profiling)

Describe alternatives you've considered

Apply the model from @Lordworms in https://github.com/apache/datafusion/pull/10166 to this pass 2. Update OptimizerRule::supports_rewrite` to return true

  1. Update OptimizerRule to use rewrite
  2. Update the pass itself to not copy the LogicalPlan (ideally using the TreeNode API) - it is implemented for LogicalPlan (API) and Expr (API)

Other examples: https://github.com/apache/datafusion/pull/10218

Additional context

alamb avatar Apr 29 '24 12:04 alamb

I am starting to unravel the remaining copies in PushDownFilter -- it is non trivial

alamb avatar May 09 '24 14:05 alamb