datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Stop copying LogicalPlan and Exprs in `ScalarSubqueryToJoin`

Open alamb opened this issue 1 year ago • 1 comments

Is your feature request related to a problem or challenge?

Part of https://github.com/apache/datafusion/issues/9637

As part of making the planner faster, we are updating the optimizer passes to avoid copying LogicalPlan and Expr (see https://github.com/apache/datafusion/issues/9637)

Describe the solution you'd like

I would like to reduce the amount of copying in this pass (even though it doesn't appear in current profiling)

Describe alternatives you've considered

Apply the model from @Lordworms in https://github.com/apache/datafusion/pull/10166 to this pass 2. Update OptimizerRule::supports_rewrite` to return true

  1. Update OptimizerRule to use rewrite
  2. Update the pass itself to not copy the LogicalPlan (ideally using the TreeNode API) - it is implemented for LogicalPlan (API) and Expr (API)

Other examples: https://github.com/apache/datafusion/pull/10218

Additional context

alamb avatar Apr 29 '24 12:04 alamb

take

rtadepalli avatar May 07 '24 02:05 rtadepalli

Since this one is a bit tricky and I don't see any draft PRs yet, I am going to take a shot at a PR as well

alamb avatar May 13 '24 16:05 alamb

Well.. I was finishing up going through the Rust book and that took more than expected because other things came up. I'll pick up another ticket, sorry about the delay.

rtadepalli avatar May 14 '24 01:05 rtadepalli

Well.. I was finishing up going through the Rust book and that took more than expected because other things came up. I'll pick up another ticket, sorry about the delay.

No worries -- I think this is a pretty tough first PR to work on. I made a PR to do it here: https://github.com/apache/datafusion/pull/10489

alamb avatar May 14 '24 19:05 alamb