starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] Support push down aggregate functions in mv rewrite

Open LiShuMing opened this issue 1 year ago • 6 comments

Why I'm doing:

For simple aggregate functions(eg: min/max/sum), we can push down the agg function even if it's not exactly matched:

    // eg:
    // sum(fn(col)) = fn(sum(col))
    // min(fn(col)) = fn(min(col))
    // max(fn(col)) = fn(max(col))

eg:

CREATE TABLE `tbl1` (
  `k1` date,
  `k2` decimal64(18, 2),
  `k3` varchar(255),
  `v1` bigint 
) ENGINE=OLAP 
DUPLICATE KEY(`k1`, `k2`, `k3`)
DISTRIBUTED BY RANDOM
PROPERTIES (
"replication_num" = "1"
);

CREATE MATERIALIZED VIEW `mv1` 
DISTRIBUTED BY RANDOM
REFRESH ASYNC
PROPERTIES (
"replication_num" = "1"
)
AS SELECT k1, k2, k3, sum(v1) from tbl1 group by k1, k2, k3

-- Cannot be rewritten by mv1!
select t1.k1, 
    sum(case when t1.k1 between date_add('2024-07-20', interval -1 month) and  date_add('2024-07-20', interval 1 month) then t1.v1 else 0 end) 
    from tbl1 t1 group by t1.k1

What I'm doing:

  • Supports to push down min/max/sum aggregate function
    • If the argument is a column ref, and operator map contains the column ref, return true
    • If the argument is a call operator and contains multi-column refs(Ony IF/CaseWhen is supported), ensure the aggregate column does not appear in the condition clause.
  • Add more tests about if(cond, val1, val2) and case when rewrite test cases.

Fixes #issue

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [ ] Yes, this PR will result in a change in behavior.
  • [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [x] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [x] 3.3
    • [ ] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

LiShuMing avatar Aug 19 '24 11:08 LiShuMing

@mergify rebase

LiShuMing avatar Aug 26 '24 05:08 LiShuMing

rebase

✅ Branch has been successfully rebased

mergify[bot] avatar Aug 26 '24 05:08 mergify[bot]

[Java-Extensions Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Sep 19 '24 16:09 github-actions[bot]

[FE Incremental Coverage Report]

:white_check_mark: pass : 104 / 112 (92.86%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/sql/optimizer/rule/transformation/materialization/EquationRewriter.java 95 103 92.23% [202, 263, 278, 286, 294, 300, 317, 336]
:large_blue_circle: com/starrocks/sql/optimizer/rule/transformation/materialization/AggregateFunctionRollupUtils.java 5 5 100.00% []
:large_blue_circle: com/starrocks/sql/analyzer/QueryAnalyzer.java 4 4 100.00% []

github-actions[bot] avatar Sep 19 '24 16:09 github-actions[bot]

[BE Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Sep 19 '24 16:09 github-actions[bot]