doris
doris copied to clipboard
apache
[opt](scanner) scan enough blocks in each scan task
Proposed changes
Follow by https://github.com/apache/doris/pull/30746, which has performance rollback in TPCH. This PR try to scan 10MB(config::doris_scanner_row_bytes) in each scheduling of scan task, restores the performance of TPCH.
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
TeamCity be ut coverage result: Function Coverage: 35.74% (8545/23907) Line Coverage: 27.55% (69330/251613) Region Coverage: 26.70% (35964/134677) Branch Coverage: 23.51% (18386/78214) Coverage Report: http://coverage.selectdb-in.cc/coverage/e84cede9cbcae0305901185d5d6a849c9d559b11_e84cede9cbcae0305901185d5d6a849c9d559b11/report/index.html
TPC-H: Total hot run time: 37676 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e84cede9cbcae0305901185d5d6a849c9d559b11, data reload: false
------ Round 1 ----------------------------------
q1 17726 4120 4051 4051
q2 2055 141 131 131
q3 10649 979 930 930
q4 4659 918 930 918
q5 7622 2882 3030 2882
q6 177 123 118 118
q7 1222 757 754 754
q8 9322 1954 2027 1954
q9 7257 6313 6283 6283
q10 8182 2493 2488 2488
q11 417 211 203 203
q12 807 310 308 308
q13 17944 3205 3217 3205
q14 276 243 254 243
q15 528 491 496 491
q16 509 392 400 392
q17 941 893 866 866
q18 6733 6076 5949 5949
q19 1566 1514 1522 1514
q20 550 265 277 265
q21 6268 3470 3560 3470
q22 801 294 261 261
Total cold run time: 106211 ms
Total hot run time: 37676 ms
----- Round 2, with runtime_filter_mode=off -----
q1 3992 3988 3980 3980
q2 308 224 214 214
q3 2963 2942 2933 2933
q4 1813 1826 1806 1806
q5 5099 5145 5107 5107
q6 197 113 116 113
q7 2147 1710 1685 1685
q8 3146 3215 3194 3194
q9 8243 8242 8207 8207
q10 6079 3653 3634 3634
q11 507 427 415 415
q12 679 550 522 522
q13 11553 3047 3040 3040
q14 280 255 253 253
q15 537 501 502 501
q16 517 454 483 454
q17 1712 1691 1679 1679
q18 7935 7460 7525 7460
q19 1651 1631 1628 1628
q20 2095 1891 1893 1891
q21 4773 4677 4578 4578
q22 497 472 429 429
Total cold run time: 66723 ms
Total hot run time: 53723 ms
TPC-DS: Total hot run time: 168418 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e84cede9cbcae0305901185d5d6a849c9d559b11, data reload: false
query1 914 339 331 331
query2 6544 1740 1665 1665
query3 6693 209 208 208
query4 23155 20627 20526 20526
query5 4313 358 353 353
query6 254 161 164 161
query7 4595 292 299 292
query8 264 203 195 195
query9 8536 2251 2230 2230
query10 409 224 208 208
query11 14690 14188 14167 14167
query12 131 84 84 84
query13 1657 441 422 422
query14 8487 6341 6338 6338
query15 199 169 177 169
query16 7079 251 252 251
query17 1054 523 496 496
query18 1891 262 260 260
query19 195 146 146 146
query20 85 81 83 81
query21 194 117 118 117
query22 4762 4566 4628 4566
query23 30480 29969 29843 29843
query24 12065 3093 3083 3083
query25 658 342 348 342
query26 1884 152 161 152
query27 3061 316 316 316
query28 7008 1819 1808 1808
query29 1212 554 542 542
query30 284 133 138 133
query31 885 687 685 685
query32 94 57 58 57
query33 709 228 230 228
query34 1065 462 480 462
query35 859 734 750 734
query36 926 872 834 834
query37 162 65 65 65
query38 3128 2967 2972 2967
query39 1299 1259 1245 1245
query40 282 101 100 100
query41 37 35 34 34
query42 99 97 98 97
query43 461 423 428 423
query44 1067 679 689 679
query45 187 183 171 171
query46 1035 810 764 764
query47 1570 1399 1401 1399
query48 405 341 342 341
query49 1160 293 289 289
query50 772 372 370 370
query51 4364 4327 4291 4291
query52 106 89 96 89
query53 331 259 261 259
query54 288 219 212 212
query55 86 78 80 78
query56 216 204 194 194
query57 959 849 841 841
query58 213 193 186 186
query59 2108 2056 1978 1978
query60 239 216 228 216
query61 82 91 81 81
query62 614 357 356 356
query63 303 255 265 255
query64 6540 2977 3124 2977
query65 3243 3210 3198 3198
query66 1443 316 311 311
query67 14688 14337 13896 13896
query68 3739 548 557 548
query69 503 395 372 372
query70 1201 1181 1201 1181
query71 309 238 266 238
query72 6010 2806 2642 2642
query73 699 315 315 315
query74 6247 6003 5965 5965
query75 3027 2436 2392 2392
query76 2333 1040 1146 1040
query77 317 232 232 232
query78 9091 8611 8436 8436
query79 1019 516 519 516
query80 680 353 344 344
query81 450 196 200 196
query82 1256 87 82 82
query83 239 124 119 119
query84 273 79 75 75
query85 1165 340 328 328
query86 299 278 291 278
query87 3250 3105 3096 3096
query88 2732 2317 2388 2317
query89 385 323 317 317
query90 2011 167 161 161
query91 152 120 117 117
query92 52 44 48 44
query93 1028 504 511 504
query94 1108 174 176 174
query95 429 332 352 332
query96 569 261 260 260
query97 4247 4108 4117 4108
query98 238 206 195 195
query99 1011 691 647 647
Total cold run time: 260847 ms
Total hot run time: 168418 ms
ClickBench: Total hot run time: 31.87 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e84cede9cbcae0305901185d5d6a849c9d559b11, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.02 0.02
query3 0.24 0.06 0.06
query4 1.67 0.10 0.10
query5 0.52 0.51 0.52
query6 1.26 0.67 0.66
query7 0.02 0.01 0.01
query8 0.03 0.02 0.03
query9 0.54 0.53 0.51
query10 0.58 0.57 0.56
query11 0.13 0.10 0.10
query12 0.12 0.10 0.10
query13 0.62 0.61 0.62
query14 0.79 0.80 0.79
query15 0.83 0.80 0.80
query16 0.37 0.39 0.38
query17 1.00 0.95 0.96
query18 0.26 0.25 0.26
query19 1.85 1.71 1.72
query20 0.02 0.01 0.01
query21 15.51 0.66 0.58
query22 2.87 4.22 2.15
query23 17.58 1.00 0.94
query24 1.99 0.77 0.37
query25 0.49 0.10 0.07
query26 0.16 0.14 0.15
query27 0.06 0.04 0.04
query28 11.91 0.83 0.82
query29 12.52 3.23 3.27
query30 0.54 0.53 0.52
query31 2.79 0.35 0.36
query32 3.35 0.48 0.49
query33 3.15 3.25 3.18
query34 15.35 4.75 4.76
query35 4.79 4.76 4.76
query36 1.13 1.06 1.07
query37 0.06 0.04 0.05
query38 0.04 0.02 0.02
query39 0.02 0.02 0.02
query40 0.16 0.14 0.12
query41 0.07 0.02 0.02
query42 0.03 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 105.54 s
Total hot run time: 31.87 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit e84cede9cbcae0305901185d5d6a849c9d559b11 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.4 seconds inserted 10000000 Rows, about 746K ops/s
TeamCity be ut coverage result: Function Coverage: 35.71% (8545/23931) Line Coverage: 27.52% (69331/251906) Region Coverage: 26.67% (35955/134820) Branch Coverage: 23.49% (18387/78292) Coverage Report: http://coverage.selectdb-in.cc/coverage/a7ecd3e46f4ab538b6171ec3190ce16155b3763d_a7ecd3e46f4ab538b6171ec3190ce16155b3763d/report/index.html
PR approved by at least one committer and no changes requested.