datafusion
datafusion copied to clipboard
Stop copying LogicalPlan and Exprs in `PushDownLimit`
Is your feature request related to a problem or challenge?
Part of https://github.com/apache/datafusion/issues/9637
As part of making the planner faster, we are updating the optimizer passes to avoid copying LogicalPlan and Expr (see https://github.com/apache/datafusion/issues/9637)
Describe the solution you'd like
I would like to reduce the amount of copying in this pass (even though it doesn't appear in current profiling)
Describe alternatives you've considered
Apply the model from @Lordworms in https://github.com/apache/datafusion/pull/10166 to this pass 2. Update OptimizerRule::supports_rewrite` to return true
- Update
OptimizerRuleto userewrite - Update the pass itself to not copy the LogicalPlan (ideally using the
TreeNodeAPI) - it is implemented forLogicalPlan(API) andExpr(API)
Other examples: https://github.com/apache/datafusion/pull/10218
Additional context
take
@kavirajk -- have you had a chance to work on this issue? If not, do you mind if someone else does?
I made https://github.com/apache/datafusion/pull/10501 to pull some of the code out into a function to maybe make this easier
@kavirajk -- have you had a chance to work on this issue? If not, do you mind if someone else does?
Sorry @alamb didn't get a chance to work on it unfortunately. Agree, feel free to work on it.
No worries. Thanks @kavirajk
I bashed out a PR for this change this morning: https://github.com/apache/datafusion/pull/10508