doris icon indicating copy to clipboard operation
doris copied to clipboard

[Enhencement](schema_scanner) Optimize the performance of reading information schema tables

Open Cai-Yao opened this issue 2 years ago • 11 comments

Proposed changes

Issue Number: close #xxx

Problem summary

  1. batch fill block
  2. batch call rpc from FE to get table desc

For 34w colunms

SELECT COUNT( * ) FROM information_schema.columns;

time: 10.3s --> 0.4s

Checklist(Required)

  • [ ] Does it affect the original behavior
  • [ ] Has unit tests been added
  • [ ] Has document been added or modified
  • [ ] Does it need to update dependencies
  • [ ] Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

Cai-Yao avatar Mar 02 '23 16:03 Cai-Yao

run buildall

Cai-Yao avatar Mar 02 '23 16:03 Cai-Yao

TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.63 seconds stream load tsv: 470 seconds loaded 74807831229 Bytes, about 151 MB/s stream load json: 37 seconds loaded 2358488459 Bytes, about 60 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230304174752_clickbench_pr_108538.html

hello-stephen avatar Mar 02 '23 17:03 hello-stephen

./run p0

yiguolei avatar Mar 03 '23 00:03 yiguolei

./run buildall

yiguolei avatar Mar 03 '23 09:03 yiguolei

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Mar 03 '23 09:03 github-actions[bot]

PR approved by anyone and no changes requested.

github-actions[bot] avatar Mar 03 '23 09:03 github-actions[bot]

./run p0

yiguolei avatar Mar 03 '23 11:03 yiguolei

run p0

Cai-Yao avatar Mar 04 '23 02:03 Cai-Yao

./run buildall

yiguolei avatar Mar 04 '23 14:03 yiguolei

run p0

Cai-Yao avatar Mar 04 '23 16:03 Cai-Yao

./run buildall

Cai-Yao avatar Mar 04 '23 17:03 Cai-Yao