doris icon indicating copy to clipboard operation
doris copied to clipboard

[feature](Nereids): add rule split limit into two phase

Open keanji-x opened this issue 2 years ago • 3 comments

Proposed changes

  • Add a rule split limit, like Limit(Origin) ==> Limit(Global) -> Gather -> Limit(Local)
  • Add a rule: limit-> sort ====> topN
  • fix a bug about topN
  • make the type of limit,offset long in topN

And because this rule is always beneficial, we add a rule in the rewrite phase

Problem summary

Describe your changes.

Checklist(Required)

  • [ ] Does it affect the original behavior
  • [x] Has unit tests been added
  • [ ] Has document been added or modified
  • [ ] Does it need to update dependencies
  • [ ] Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

keanji-x avatar Feb 15 '23 12:02 keanji-x

TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.76 seconds stream load tsv: 480 seconds loaded 74807831229 Bytes, about 148 MB/s stream load json: 40 seconds loaded 2358488459 Bytes, about 56 MB/s stream load orc: 74 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230307060535_clickbench_pr_109806.html

hello-stephen avatar Feb 15 '23 14:02 hello-stephen

run buildall

keanji-x avatar Mar 01 '23 01:03 keanji-x

run buildall

keanji-x avatar Mar 02 '23 04:03 keanji-x

run buildall

keanji-x avatar Mar 06 '23 02:03 keanji-x

run buildall

keanji-x avatar Mar 07 '23 03:03 keanji-x

run buildall

keanji-x avatar Mar 07 '23 05:03 keanji-x

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Mar 07 '23 07:03 github-actions[bot]

PR approved by anyone and no changes requested.

github-actions[bot] avatar Mar 07 '23 07:03 github-actions[bot]