datafusion
datafusion copied to clipboard
Stop copying LogicalPlan and Exprs in `EliminateCrossJoin` (4% faster planning)
Draft as it builds on
- https://github.com/apache/datafusion/pull/10410
- https://github.com/apache/datafusion/pull/10427
- https://github.com/apache/datafusion/pull/10430
Which issue does this PR close?
Closes https://github.com/apache/datafusion/issues/10287
Rationale for this change
Make planning faster by avoiding copying
What changes are included in this PR?
- Change
EliminateCrossJointo use TreeNode API and stop copying plans - Reduce
Exprcopies more - Improve documentation
Are these changes tested?
Existing CI
Are there any user-facing changes?
No functional changes,
Performance change: 4% better planning
(note this includes the improvements from https://github.com/apache/datafusion/pull/10427 as well)
Details
group eliminate_cross_join main
----- -------------------- ----
logical_aggregate_with_join 1.00 1214.6±15.54µs ? ?/sec 1.00 1218.8±14.06µs ? ?/sec
logical_plan_tpcds_all 1.01 161.6±1.93ms ? ?/sec 1.00 159.4±1.84ms ? ?/sec
logical_plan_tpch_all 1.02 17.2±0.19ms ? ?/sec 1.00 16.9±0.16ms ? ?/sec
logical_select_all_from_1000 1.00 18.0±0.08ms ? ?/sec 1.05 18.8±0.09ms ? ?/sec
logical_select_one_from_700 1.00 822.4±15.44µs ? ?/sec 1.00 820.4±44.09µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.01 765.5±9.28µs ? ?/sec 1.00 759.0±13.89µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.01 751.8±9.19µs ? ?/sec 1.00 741.7±23.29µs ? ?/sec
physical_plan_tpcds_all 1.00 1310.5±5.75ms ? ?/sec 1.04 1357.6±7.25ms ? ?/sec
physical_plan_tpch_all 1.00 90.1±0.88ms ? ?/sec 1.04 94.1±0.82ms ? ?/sec
physical_plan_tpch_q1 1.00 3.9±0.05ms ? ?/sec 1.33 5.2±0.07ms ? ?/sec
physical_plan_tpch_q10 1.00 4.3±0.06ms ? ?/sec 1.03 4.4±0.07ms ? ?/sec
physical_plan_tpch_q11 1.00 3.9±0.06ms ? ?/sec 1.03 4.0±0.05ms ? ?/sec
physical_plan_tpch_q12 1.00 3.0±0.05ms ? ?/sec 1.02 3.1±0.07ms ? ?/sec
physical_plan_tpch_q13 1.00 2.1±0.03ms ? ?/sec 1.00 2.2±0.04ms ? ?/sec
physical_plan_tpch_q14 1.00 2.5±0.04ms ? ?/sec 1.10 2.8±0.04ms ? ?/sec
physical_plan_tpch_q16 1.00 3.6±0.04ms ? ?/sec 1.07 3.9±0.04ms ? ?/sec
physical_plan_tpch_q17 1.00 3.5±0.05ms ? ?/sec 1.04 3.7±0.05ms ? ?/sec
physical_plan_tpch_q18 1.00 3.9±0.07ms ? ?/sec 1.03 4.1±0.04ms ? ?/sec
physical_plan_tpch_q19 1.00 5.9±0.05ms ? ?/sec 1.08 6.4±0.05ms ? ?/sec
physical_plan_tpch_q2 1.00 7.7±0.06ms ? ?/sec 1.03 7.9±0.05ms ? ?/sec
physical_plan_tpch_q20 1.00 4.6±0.06ms ? ?/sec 1.02 4.7±0.06ms ? ?/sec
physical_plan_tpch_q21 1.00 6.2±0.08ms ? ?/sec 1.02 6.3±0.05ms ? ?/sec
physical_plan_tpch_q22 1.00 3.4±0.06ms ? ?/sec 1.04 3.5±0.05ms ? ?/sec
physical_plan_tpch_q3 1.00 3.1±0.04ms ? ?/sec 1.07 3.3±0.04ms ? ?/sec
physical_plan_tpch_q4 1.00 2.3±0.03ms ? ?/sec 1.04 2.4±0.06ms ? ?/sec
physical_plan_tpch_q5 1.00 4.4±0.06ms ? ?/sec 1.05 4.7±0.05ms ? ?/sec
physical_plan_tpch_q6 1.00 1507.9±41.94µs ? ?/sec 1.06 1598.3±24.25µs ? ?/sec
physical_plan_tpch_q7 1.00 5.7±0.06ms ? ?/sec 1.03 5.8±0.07ms ? ?/sec
physical_plan_tpch_q8 1.00 7.3±0.05ms ? ?/sec 1.03 7.5±0.07ms ? ?/sec
physical_plan_tpch_q9 1.00 5.6±0.07ms ? ?/sec 1.02 5.7±0.07ms ? ?/sec
physical_select_all_from_1000 1.00 58.7±0.25ms ? ?/sec 1.05 61.5±1.01ms ? ?/sec
physical_select_one_from_700 1.00 3.7±0.03ms ? ?/sec 1.02 3.7±0.05ms ? ?/sec