datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Stop copying LogicalPlan and Exprs in `EliminateCrossJoin`

Open alamb opened this issue 1 year ago • 3 comments

Is your feature request related to a problem or challenge?

Part of https://github.com/apache/datafusion/issues/9637

As part of making the planner faster, we are updating the optimizer passes to avoid copying LogicalPlan and Expr (see https://github.com/apache/datafusion/issues/9637)

Describe the solution you'd like

I would like to reduce the amount of copying in this pass (even though it doesn't appear in current profiling)

Describe alternatives you've considered

Apply the model from @Lordworms in https://github.com/apache/datafusion/pull/10166 to this pass 2. Update OptimizerRule::supports_rewrite` to return true

  1. Update OptimizerRule to use rewrite
  2. Update the pass itself to not copy the LogicalPlan (ideally using the TreeNode API) - it is implemented for LogicalPlan (API) and Expr (API)

Other examples: https://github.com/apache/datafusion/pull/10218

Additional context

alamb avatar Apr 29 '24 12:04 alamb

I believe @Lordworms is working on this -- https://github.com/apache/datafusion/issues/9637#issuecomment-2075311002

alamb avatar Apr 29 '24 12:04 alamb

I am going to give this one a try

alamb avatar May 08 '24 18:05 alamb

It is done over a few PRs but I have this change now working and I think it is looking quite good: https://github.com/apache/datafusion/pull/10431

alamb avatar May 09 '24 13:05 alamb