doris
doris copied to clipboard
apache
[PipelineX](improvement) Prepare tasks in parallel
Proposed changes
Now pipeline tasks are prepared in serial mode which has high overhead for short queries. This PR change it to parallel preparation.
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
TeamCity be ut coverage result: Function Coverage: 35.25% (8732/24770) Line Coverage: 27.03% (71503/264501) Region Coverage: 26.29% (37098/141135) Branch Coverage: 23.18% (18969/81842) Coverage Report: http://coverage.selectdb-in.cc/coverage/dcf772d4ab553c4f7805fad0221ae5aa57267747_dcf772d4ab553c4f7805fad0221ae5aa57267747/report/index.html
TeamCity be ut coverage result: Function Coverage: 35.25% (8732/24770) Line Coverage: 27.04% (71526/264501) Region Coverage: 26.29% (37100/141137) Branch Coverage: 23.19% (18977/81844) Coverage Report: http://coverage.selectdb-in.cc/coverage/b2b1a8fd463b5ccd4aff4a269f31a23b79744f4f_b2b1a8fd463b5ccd4aff4a269f31a23b79744f4f/report/index.html
TeamCity be ut coverage result: Function Coverage: 35.25% (8735/24777) Line Coverage: 27.03% (71531/264661) Region Coverage: 26.28% (37107/141188) Branch Coverage: 23.17% (18975/81884) Coverage Report: http://coverage.selectdb-in.cc/coverage/62324e15b00b1d474b0e1ef47759ecdce5d79221_62324e15b00b1d474b0e1ef47759ecdce5d79221/report/index.html
TeamCity be ut coverage result: Function Coverage: 35.26% (8737/24782) Line Coverage: 27.02% (71524/264733) Region Coverage: 26.27% (37114/141260) Branch Coverage: 23.17% (18975/81908) Coverage Report: http://coverage.selectdb-in.cc/coverage/62324e15b00b1d474b0e1ef47759ecdce5d79221_62324e15b00b1d474b0e1ef47759ecdce5d79221/report/index.html
TeamCity be ut coverage result: Function Coverage: 35.60% (8904/25011) Line Coverage: 27.30% (73092/267776) Region Coverage: 26.44% (37797/142963) Branch Coverage: 23.20% (19264/83048) Coverage Report: http://coverage.selectdb-in.cc/coverage/36d531d736fc8d54873f82a3bbef5f4ad8ae8731_36d531d736fc8d54873f82a3bbef5f4ad8ae8731/report/index.html
TeamCity be ut coverage result: Function Coverage: 35.42% (8920/25184) Line Coverage: 27.13% (73289/270140) Region Coverage: 26.25% (37835/144124) Branch Coverage: 23.06% (19277/83586) Coverage Report: http://coverage.selectdb-in.cc/coverage/6047d6d3ee97a21b3d36461df074d3daababe733_6047d6d3ee97a21b3d36461df074d3daababe733/report/index.html
TeamCity be ut coverage result: Function Coverage: 35.42% (8920/25184) Line Coverage: 27.13% (73288/270145) Region Coverage: 26.25% (37838/144126) Branch Coverage: 23.07% (19284/83588) Coverage Report: http://coverage.selectdb-in.cc/coverage/e7c9d215f45eff0d56edb5487c5f00a85f3696af_e7c9d215f45eff0d56edb5487c5f00a85f3696af/report/index.html
TPC-H: Total hot run time: 38130 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a018575f4cf3da1a5f4d7309606077cdb9e9191b, data reload: false
------ Round 1 ----------------------------------
q1 17612 4352 4229 4229
q2 2006 183 173 173
q3 10454 1226 1203 1203
q4 10196 811 751 751
q5 7535 2612 2627 2612
q6 213 130 131 130
q7 1003 595 572 572
q8 9220 2034 2020 2020
q9 7317 6598 6512 6512
q10 8529 3487 3529 3487
q11 430 222 212 212
q12 462 211 207 207
q13 18940 2916 2932 2916
q14 260 223 229 223
q15 531 484 461 461
q16 523 370 370 370
q17 955 633 679 633
q18 7385 6661 6613 6613
q19 5889 1545 1475 1475
q20 625 304 291 291
q21 3428 2779 2744 2744
q22 353 296 308 296
Total cold run time: 113866 ms
Total hot run time: 38130 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4400 4217 4224 4217
q2 358 252 260 252
q3 2958 2670 2739 2670
q4 1863 1596 1536 1536
q5 5329 5272 5304 5272
q6 203 122 122 122
q7 2207 1852 1839 1839
q8 3197 3321 3292 3292
q9 8562 8574 8548 8548
q10 4071 3917 3953 3917
q11 636 485 496 485
q12 820 620 592 592
q13 16317 3230 3205 3205
q14 323 284 274 274
q15 501 471 490 471
q16 485 415 445 415
q17 1823 1531 1535 1531
q18 8126 7767 7803 7767
q19 1731 1558 1539 1539
q20 1944 1797 1866 1797
q21 5196 4881 4895 4881
q22 521 458 452 452
Total cold run time: 71571 ms
Total hot run time: 55074 ms
TeamCity be ut coverage result: Function Coverage: 35.42% (8922/25188) Line Coverage: 27.10% (73276/270381) Region Coverage: 26.25% (37852/144223) Branch Coverage: 23.05% (19279/83632) Coverage Report: http://coverage.selectdb-in.cc/coverage/a018575f4cf3da1a5f4d7309606077cdb9e9191b_a018575f4cf3da1a5f4d7309606077cdb9e9191b/report/index.html
TPC-DS: Total hot run time: 183049 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a018575f4cf3da1a5f4d7309606077cdb9e9191b, data reload: false
query1 884 362 345 345
query2 6704 2493 2415 2415
query3 6639 199 195 195
query4 23584 21314 21215 21215
query5 4094 379 376 376
query6 253 167 161 161
query7 4565 274 280 274
query8 233 183 170 170
query9 8398 2287 2243 2243
query10 405 219 231 219
query11 14767 14180 14113 14113
query12 128 82 76 76
query13 1626 349 347 347
query14 9320 7062 7033 7033
query15 213 177 177 177
query16 7860 247 250 247
query17 1944 587 572 572
query18 1951 274 272 272
query19 194 144 150 144
query20 85 76 82 76
query21 200 131 117 117
query22 5044 4804 4815 4804
query23 33530 32807 32862 32807
query24 11185 3008 3062 3008
query25 589 385 377 377
query26 684 154 151 151
query27 2465 366 356 356
query28 5657 2046 2038 2038
query29 888 626 608 608
query30 328 172 170 170
query31 983 759 734 734
query32 96 49 50 49
query33 619 224 227 224
query34 1024 490 493 490
query35 846 720 701 701
query36 1088 913 916 913
query37 101 68 67 67
query38 3528 3391 3299 3299
query39 1661 1586 1572 1572
query40 173 122 118 118
query41 50 45 41 41
query42 102 93 96 93
query43 531 531 525 525
query44 1152 731 755 731
query45 282 254 262 254
query46 1130 777 739 739
query47 2024 1957 1933 1933
query48 360 285 301 285
query49 851 384 385 384
query50 775 391 388 388
query51 6875 6809 6944 6809
query52 105 85 87 85
query53 344 271 273 271
query54 302 218 225 218
query55 141 69 68 68
query56 216 194 201 194
query57 1294 1197 1183 1183
query58 194 178 184 178
query59 3531 3439 3208 3208
query60 236 221 197 197
query61 71 71 87 71
query62 590 423 439 423
query63 292 264 262 262
query64 4728 2870 3744 2870
query65 3036 2984 2972 2972
query66 728 301 307 301
query67 15637 14981 14907 14907
query68 9796 581 541 541
query69 557 285 278 278
query70 1416 1140 1127 1127
query71 1462 1262 1257 1257
query72 6853 2598 2395 2395
query73 1586 308 314 308
query74 6843 6419 6445 6419
query75 4724 2534 2604 2534
query76 5417 985 951 951
query77 593 229 232 229
query78 11006 10142 10174 10142
query79 10311 512 508 508
query80 2060 390 399 390
query81 502 223 227 223
query82 162 85 82 82
query83 198 137 135 135
query84 259 75 73 73
query85 1073 247 240 240
query86 330 256 274 256
query87 3487 3285 3223 3223
query88 5870 2263 2269 2263
query89 474 355 349 349
query90 2451 166 164 164
query91 112 83 86 83
query92 51 42 47 42
query93 5475 505 505 505
query94 1574 166 163 163
query95 1087 1114 1080 1080
query96 607 263 253 253
query97 3102 2918 2965 2918
query98 226 217 207 207
query99 1081 853 848 848
Total cold run time: 300858 ms
Total hot run time: 183049 ms
ClickBench: Total hot run time: 30.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a018575f4cf3da1a5f4d7309606077cdb9e9191b, data reload: false
query1 0.03 0.03 0.03
query2 0.08 0.04 0.04
query3 0.23 0.04 0.04
query4 1.69 0.09 0.10
query5 0.53 0.51 0.49
query6 1.46 0.73 0.71
query7 0.02 0.01 0.02
query8 0.05 0.04 0.03
query9 0.54 0.49 0.49
query10 0.54 0.55 0.54
query11 0.15 0.12 0.12
query12 0.14 0.12 0.11
query13 0.60 0.59 0.58
query14 0.74 0.77 0.76
query15 0.84 0.80 0.80
query16 0.36 0.35 0.34
query17 1.00 0.94 0.99
query18 0.21 0.23 0.24
query19 1.77 1.71 1.71
query20 0.01 0.01 0.01
query21 15.40 0.66 0.65
query22 4.97 6.86 1.73
query23 18.33 1.40 1.39
query24 1.96 0.24 0.21
query25 0.16 0.08 0.08
query26 0.27 0.16 0.17
query27 0.07 0.07 0.08
query28 13.26 1.01 0.99
query29 12.98 3.26 3.26
query30 0.26 0.09 0.06
query31 2.83 0.38 0.36
query32 3.32 0.46 0.45
query33 2.80 2.84 2.85
query34 17.16 4.42 4.43
query35 4.43 4.46 4.48
query36 0.63 0.46 0.44
query37 0.18 0.15 0.15
query38 0.15 0.15 0.14
query39 0.05 0.03 0.03
query40 0.17 0.12 0.13
query41 0.08 0.04 0.05
query42 0.06 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 110.55 s
Total hot run time: 30.21 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit a018575f4cf3da1a5f4d7309606077cdb9e9191b with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s