doris icon indicating copy to clipboard operation
doris copied to clipboard

[feature](nereids)support bitmap runtime filter on nereids

Open englefly opened this issue 2 years ago • 3 comments

Proposed changes

select a from t where a in (select k from bitmap_table) , in which t.a is int, and bitmap_table.k is bitmap this sql is transformed to select a from t left semi join (select bitmap_union(k2) u from bitmap_table) t2 on bitmap_contains(t2.u, t.a) and then we generate runtime filter for expression bitmap_contains(t2.u, t.a)

original planner only support in-subquery

Issue Number: close #xxx

Problem summary

Describe your changes.

Checklist(Required)

  • [ ] Does it affect the original behavior
  • [ ] Has unit tests been added
  • [ ] Has document been added or modified
  • [ ] Does it need to update dependencies
  • [ ] Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

englefly avatar Feb 20 '23 04:02 englefly

TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.62 seconds stream load tsv: 455 seconds loaded 74807831229 Bytes, about 156 MB/s stream load json: 40 seconds loaded 2358488459 Bytes, about 56 MB/s stream load orc: 74 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230308141156_clickbench_pr_110905.html

hello-stephen avatar Feb 20 '23 08:02 hello-stephen

run buildall

morrySnow avatar Feb 24 '23 07:02 morrySnow

run buildall

englefly avatar Mar 02 '23 03:03 englefly

run buildall

englefly avatar Mar 05 '23 07:03 englefly

run buildall

englefly avatar Mar 06 '23 10:03 englefly

run buildall

morrySnow avatar Mar 07 '23 06:03 morrySnow

run buildall

englefly avatar Mar 07 '23 11:03 englefly

run buildall

englefly avatar Mar 08 '23 13:03 englefly

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Mar 09 '23 01:03 github-actions[bot]

PR approved by anyone and no changes requested.

github-actions[bot] avatar Mar 09 '23 01:03 github-actions[bot]