starrocks
starrocks copied to clipboard
[Enhancement] Move cte inline to memo phase 1
What type of PR is this:
- [ ] bug
- [ ] feature
- [x] enhancement
- [ ] refactor
- [ ] others
Which issues of this PR fixes :
Fixes #
Problem Summary(Required) :
Move cte inline rule to memo phase, we will enforce all inline or all reuse cte
- add CTENoOP implements
- add CTEProperty in Enforce phase
- add CTE costs model
Opimizer Performance:
| Query | Before | After |
|---|---|---|
| multi-nest view query with cte(8+) #9347 | 300ms | 300ms |
| 20 iterate cte + 60 refs | 100ms | - |
| 12 iterate cte + 36 refs | - | 2224ms |
| 11 iterate cte + 33 refs | - | 700ms |
| 10 iterate cte + 30 refs | - | 280ms |
| 8 nested cte + 3 * 2^8 refs | 4081ms | 7522ms |
| 7 nested cte + 3 * 2^7 refs | 1481ms | 2250ms |
| 6 nested cte + 3 * 2^6 refs | 451ms | 600ms |
Of course, the cost of these test is in RuleWrite phase, not in memo phase.It's will take more time if we test so many joins or complex SQL
BTW, we need optimize the cost of memo merge
SQLs
# iterater cte
with x1 as (select * from t0),
x2 as (select * from t0),
x3 as (select * from t0),
x4 as (select * from t0),
x5 as (select * from t0),
x6 as (select * from t0),
x7 as (select * from t0),
x8 as (select * from t0),
x9 as (select * from t0),
x10 as (select * from t0),
x11 as (select * from t0)
select * from x1 union all
select * from x1 union all
select * from x1 union all
select * from x2 union all
select * from x2 union all
select * from x2 union all
select * from x3 union all
select * from x3 union all
select * from x3 union all
select * from x4 union all
select * from x4 union all
select * from x4 union all
select * from x5 union all
select * from x5 union all
select * from x5 union all
select * from x6 union all
select * from x6 union all
select * from x6 union all
select * from x7 union all
select * from x7 union all
select * from x7 union all
select * from x8 union all
select * from x8 union all
select * from x8 union all
select * from x9 union all
select * from x9 union all
select * from x9 union all
select * from x10 union all
select * from x10 union all
select * from x10 union all
select * from x11 union all
select * from x11 union all
select * from x11;
# nest cte
explain
with x1 as (select * from t0),
x2 as (select * from x1 union all
select * from x1),
x3 as (select * from x2 union all
select * from x2),
x4 as (select * from x3 union all
select * from x3),
x5 as (select * from x4 union all
select * from x4),
x6 as (select * from x5 union all
select * from x5),
x7 as (select * from x6 union all
select * from x6)
select * from x1 union all
select * from x1 union all
select * from x1 union all
select * from x2 union all
select * from x2 union all
select * from x2 union all
select * from x3 union all
select * from x3 union all
select * from x3 union all
select * from x4 union all
select * from x4 union all
select * from x4 union all
select * from x5 union all
select * from x5 union all
select * from x5 union all
select * from x6 union all
select * from x6 union all
select * from x6 union all
select * from x7 union all
select * from x7 union all
select * from x7 ;
Need the optimizer itself performance test, we need to know the optimizer will slow how much time.
run starrocks_fe_unittest
run starrocks_admit_test
run starrocks_admit_test
run starrocks_fe_unittest
run starrocks_admit_test
run starrocks_fe_unittest
run starrocks_admit_test
run starrocks_fe_unittest
[FE PR Coverage Check]
:heart_eyes: pass : 190 / 208 (91.35%)
file detail
| path | covered_line | new_line | coverage | not_covered_line_detail | |
|---|---|---|---|---|---|
| :large_blue_circle: | com/starrocks/sql/optimizer/operator/logical/LogicalCTEConsumeOperator.java | 0 | 1 | 00.00% | [84] |
| :large_blue_circle: | com/starrocks/sql/optimizer/rule/transformation/InlineCTEConsumeRule.java | 0 | 4 | 00.00% | [14, 17, 21, 22] |
| :large_blue_circle: | com/starrocks/sql/optimizer/operator/physical/PhysicalCTEAnchorOperator.java | 1 | 2 | 50.00% | [29] |
| :large_blue_circle: | com/starrocks/sql/optimizer/operator/physical/PhysicalNoCTEOperator.java | 4 | 5 | 80.00% | [43] |
| :large_blue_circle: | com/starrocks/sql/optimizer/base/CTEProperty.java | 24 | 29 | 82.76% | [54, 59, 60, 69, 82] |
| :large_blue_circle: | com/starrocks/sql/optimizer/ChildOutputPropertyGuarantor.java | 27 | 30 | 90.00% | [148, 149, 150] |
| :large_blue_circle: | com/starrocks/sql/optimizer/statistics/StatisticsCalculator.java | 12 | 13 | 92.31% | [1372] |
| :large_blue_circle: | com/starrocks/sql/optimizer/task/EnforceAndCostTask.java | 14 | 15 | 93.33% | [215] |
| :large_blue_circle: | com/starrocks/sql/optimizer/base/PhysicalPropertySet.java | 14 | 15 | 93.33% | [97] |
| :large_blue_circle: | com/starrocks/sql/optimizer/RequiredPropertyDeriver.java | 27 | 27 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/optimizer/Optimizer.java | 5 | 5 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/optimizer/rule/implementation/CTEConsumerReuseImplementationRule.java | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/optimizer/rule/implementation/CTEAnchorToNoCTEImplementationRule.java | 2 | 2 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/optimizer/rule/implementation/CTEConsumeInlineImplementationRule.java | 8 | 8 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/optimizer/rule/implementation/CTEAnchorImplementationRule.java | 3 | 3 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/optimizer/rule/RuleType.java | 2 | 2 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/optimizer/OutputPropertyDeriver.java | 24 | 24 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/optimizer/rule/transformation/InlineOneCTEConsumeRule.java | 6 | 6 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/optimizer/CTEContext.java | 8 | 8 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/optimizer/CTEUtils.java | 4 | 4 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/optimizer/cost/CostModel.java | 4 | 4 | 100.00% | [] |
run starrocks_admit_test







