starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] Move cte inline to memo phase 1

Open Seaven opened this issue 3 years ago • 3 comments

What type of PR is this:

  • [ ] bug
  • [ ] feature
  • [x] enhancement
  • [ ] refactor
  • [ ] others

Which issues of this PR fixes :

Fixes #

Problem Summary(Required) :

Move cte inline rule to memo phase, we will enforce all inline or all reuse cte

  1. add CTENoOP implements
  2. add CTEProperty in Enforce phase
  3. add CTE costs model

Opimizer Performance:

Query Before After
multi-nest view query with cte(8+) #9347 300ms 300ms
20 iterate cte + 60 refs 100ms -
12 iterate cte + 36 refs - 2224ms
11 iterate cte + 33 refs - 700ms
10 iterate cte + 30 refs - 280ms
8 nested cte + 3 * 2^8 refs 4081ms 7522ms
7 nested cte + 3 * 2^7 refs 1481ms 2250ms
6 nested cte + 3 * 2^6 refs 451ms 600ms

Of course, the cost of these test is in RuleWrite phase, not in memo phase.It's will take more time if we test so many joins or complex SQL

BTW, we need optimize the cost of memo merge

SQLs

# iterater cte
with x1 as (select * from t0),
     x2 as (select * from t0),
     x3 as (select * from t0),
     x4 as (select * from t0),
     x5 as (select * from t0),
     x6 as (select * from t0),                    
     x7 as (select * from t0),
     x8 as (select * from t0),          
     x9 as (select * from t0),          
     x10 as (select * from t0),
     x11 as (select * from t0)     
select * from x1 union all
select * from x1 union all
select * from x1 union all
select * from x2 union all
select * from x2 union all
select * from x2 union all
select * from x3 union all
select * from x3 union all
select * from x3 union all
select * from x4 union all
select * from x4 union all
select * from x4 union all
select * from x5 union all
select * from x5 union all
select * from x5 union all
select * from x6 union all
select * from x6 union all
select * from x6 union all
select * from x7 union all
select * from x7 union all
select * from x7 union all
select * from x8 union all
select * from x8 union all
select * from x8 union all
select * from x9 union all
select * from x9 union all
select * from x9 union all
select * from x10 union all
select * from x10 union all
select * from x10 union all
select * from x11 union all
select * from x11 union all
select * from x11;

# nest cte
explain 
with x1 as (select * from t0),
     x2 as (select * from x1 union all 
            select * from x1),
     x3 as (select * from x2 union all 
            select * from x2),
     x4 as (select * from x3 union all 
            select * from x3),
     x5 as (select * from x4 union all 
            select * from x4),
     x6 as (select * from x5 union all 
            select * from x5),
     x7 as (select * from x6 union all 
            select * from x6)
select * from x1 union all
select * from x1 union all
select * from x1 union all
select * from x2 union all
select * from x2 union all
select * from x2 union all
select * from x3 union all
select * from x3 union all
select * from x3 union all
select * from x4 union all
select * from x4 union all
select * from x4 union all
select * from x5 union all
select * from x5 union all
select * from x5 union all
select * from x6 union all
select * from x6 union all
select * from x6 union all
select * from x7 union all
select * from x7 union all
select * from x7 ;


Seaven avatar Aug 04 '22 03:08 Seaven

Need the optimizer itself performance test, we need to know the optimizer will slow how much time.

kangkaisen avatar Aug 08 '22 13:08 kangkaisen

run starrocks_fe_unittest

Seaven avatar Aug 15 '22 07:08 Seaven

run starrocks_admit_test

wanpengfei-git avatar Aug 15 '22 12:08 wanpengfei-git

run starrocks_admit_test

wanpengfei-git avatar Aug 16 '22 07:08 wanpengfei-git

run starrocks_fe_unittest

Seaven avatar Aug 16 '22 12:08 Seaven

run starrocks_admit_test

wanpengfei-git avatar Aug 17 '22 02:08 wanpengfei-git

run starrocks_fe_unittest

Seaven avatar Aug 17 '22 02:08 Seaven

run starrocks_admit_test

wanpengfei-git avatar Aug 17 '22 03:08 wanpengfei-git

run starrocks_fe_unittest

Seaven avatar Aug 17 '22 05:08 Seaven

[FE PR Coverage Check]

:heart_eyes: pass : 190 / 208 (91.35%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/sql/optimizer/operator/logical/LogicalCTEConsumeOperator.java 0 1 00.00% [84]
:large_blue_circle: com/starrocks/sql/optimizer/rule/transformation/InlineCTEConsumeRule.java 0 4 00.00% [14, 17, 21, 22]
:large_blue_circle: com/starrocks/sql/optimizer/operator/physical/PhysicalCTEAnchorOperator.java 1 2 50.00% [29]
:large_blue_circle: com/starrocks/sql/optimizer/operator/physical/PhysicalNoCTEOperator.java 4 5 80.00% [43]
:large_blue_circle: com/starrocks/sql/optimizer/base/CTEProperty.java 24 29 82.76% [54, 59, 60, 69, 82]
:large_blue_circle: com/starrocks/sql/optimizer/ChildOutputPropertyGuarantor.java 27 30 90.00% [148, 149, 150]
:large_blue_circle: com/starrocks/sql/optimizer/statistics/StatisticsCalculator.java 12 13 92.31% [1372]
:large_blue_circle: com/starrocks/sql/optimizer/task/EnforceAndCostTask.java 14 15 93.33% [215]
:large_blue_circle: com/starrocks/sql/optimizer/base/PhysicalPropertySet.java 14 15 93.33% [97]
:large_blue_circle: com/starrocks/sql/optimizer/RequiredPropertyDeriver.java 27 27 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/Optimizer.java 5 5 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/rule/implementation/CTEConsumerReuseImplementationRule.java 1 1 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/rule/implementation/CTEAnchorToNoCTEImplementationRule.java 2 2 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/rule/implementation/CTEConsumeInlineImplementationRule.java 8 8 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/rule/implementation/CTEAnchorImplementationRule.java 3 3 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/rule/RuleType.java 2 2 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/OutputPropertyDeriver.java 24 24 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/rule/transformation/InlineOneCTEConsumeRule.java 6 6 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/CTEContext.java 8 8 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/CTEUtils.java 4 4 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/cost/CostModel.java 4 4 100.00% []

wanpengfei-git avatar Aug 17 '22 05:08 wanpengfei-git

run starrocks_admit_test

Seaven avatar Aug 17 '22 05:08 Seaven