doris
doris copied to clipboard
apache
[improve](routine-load) optimize routine load task schedule to make consume real-time and balanced
Proposed changes
If be slot is full, the task will be put in the end, causing the partition corresponding to this task will be delayed in consumption and consumption not balanced.
Therefore, the PR keeps the task still in the head of the queue to avoid it.
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
TPC-H: Total hot run time: 41282 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e1cac9c6c23948c819b613d94c140b489648bc87, data reload: false
------ Round 1 ----------------------------------
q1 17689 5009 4954 4954
q2 2042 140 132 132
q3 10587 994 980 980
q4 4652 969 974 969
q5 7626 3162 3213 3162
q6 193 134 131 131
q7 1259 778 765 765
q8 9270 2094 2078 2078
q9 7511 6619 6625 6619
q10 8302 2651 2643 2643
q11 410 217 204 204
q12 770 330 329 329
q13 17935 3615 3599 3599
q14 293 256 269 256
q15 588 521 502 502
q16 475 400 411 400
q17 933 890 860 860
q18 7414 6628 6624 6624
q19 1541 1500 1505 1500
q20 559 275 272 272
q21 6389 3982 3967 3967
q22 863 337 336 336
Total cold run time: 107301 ms
Total hot run time: 41282 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4913 4852 4869 4852
q2 296 184 184 184
q3 3623 3608 3621 3608
q4 2540 2525 2530 2525
q5 5792 5800 5805 5800
q6 215 127 126 126
q7 2287 1625 1681 1625
q8 3072 3120 3170 3120
q9 8797 8769 8780 8769
q10 6920 4240 4240 4240
q11 515 372 382 372
q12 774 543 545 543
q13 4177 3393 3418 3393
q14 273 252 231 231
q15 593 547 530 530
q16 488 448 425 425
q17 1700 1628 1627 1627
q18 8408 7684 7639 7639
q19 1642 1644 1645 1644
q20 2124 1839 1827 1827
q21 6604 6215 6214 6214
q22 566 503 504 503
Total cold run time: 66319 ms
Total hot run time: 59797 ms
TPC-DS: Total hot run time: 176630 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e1cac9c6c23948c819b613d94c140b489648bc87, data reload: false
query1 916 353 353 353
query2 6507 1775 1767 1767
query3 6692 210 207 207
query4 23103 21153 21083 21083
query5 4282 373 379 373
query6 257 166 169 166
query7 4609 296 291 291
query8 249 200 195 195
query9 8423 2841 2815 2815
query10 418 229 218 218
query11 15127 14481 14486 14481
query12 145 82 83 82
query13 1684 416 411 411
query14 9176 7642 7520 7520
query15 209 188 190 188
query16 7476 256 243 243
query17 1428 559 531 531
query18 1944 271 262 262
query19 204 145 149 145
query20 89 82 85 82
query21 186 120 124 120
query22 5049 4994 5012 4994
query23 32506 31457 31571 31457
query24 12744 3458 3379 3379
query25 640 360 357 357
query26 1880 163 164 163
query27 3041 315 313 313
query28 6622 1870 1859 1859
query29 1157 608 618 608
query30 281 136 143 136
query31 963 754 774 754
query32 98 58 55 55
query33 723 222 242 222
query34 1087 489 505 489
query35 937 823 821 821
query36 952 926 854 854
query37 182 65 59 59
query38 3292 3125 3214 3125
query39 1361 1336 1314 1314
query40 284 105 107 105
query41 37 33 35 33
query42 103 102 99 99
query43 479 441 446 441
query44 1069 679 707 679
query45 199 178 173 173
query46 1065 777 778 777
query47 1698 1579 1528 1528
query48 461 347 335 335
query49 1214 304 299 299
query50 776 367 367 367
query51 4460 4297 4341 4297
query52 106 92 97 92
query53 392 305 306 305
query54 297 217 226 217
query55 86 82 81 81
query56 226 208 198 198
query57 1068 986 967 967
query58 220 193 196 193
query59 2252 2240 2112 2112
query60 238 223 208 208
query61 84 89 82 82
query62 610 386 381 381
query63 320 281 287 281
query64 6501 3047 3113 3047
query65 3288 3266 3268 3266
query66 1351 324 318 318
query67 14535 14480 14217 14217
query68 5110 567 541 541
query69 497 361 355 355
query70 1249 1264 1195 1195
query71 377 251 271 251
query72 6311 2788 2622 2622
query73 696 309 311 309
query74 6823 6406 6363 6363
query75 3197 2559 2568 2559
query76 3056 1129 1176 1129
query77 356 242 231 231
query78 9381 8897 8795 8795
query79 983 519 500 500
query80 509 355 343 343
query81 442 206 210 206
query82 162 83 81 81
query83 137 121 120 120
query84 229 78 76 76
query85 1026 342 338 338
query86 302 319 307 307
query87 3488 3303 3311 3303
query88 2684 2280 2285 2280
query89 439 345 342 342
query90 1845 167 165 165
query91 157 127 129 127
query92 52 49 48 48
query93 1011 520 505 505
query94 1154 177 176 176
query95 440 338 335 335
query96 583 264 264 264
query97 4441 4288 4256 4256
query98 222 209 189 189
query99 1029 787 729 729
Total cold run time: 268446 ms
Total hot run time: 176630 ms
ClickBench: Total hot run time: 31.48 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e1cac9c6c23948c819b613d94c140b489648bc87, data reload: false
query1 0.02 0.02 0.02
query2 0.06 0.02 0.02
query3 0.22 0.06 0.07
query4 1.66 0.09 0.07
query5 0.49 0.48 0.48
query6 1.33 0.61 0.63
query7 0.02 0.01 0.02
query8 0.04 0.03 0.03
query9 0.54 0.47 0.46
query10 0.49 0.50 0.51
query11 0.12 0.09 0.10
query12 0.12 0.09 0.10
query13 0.58 0.58 0.59
query14 0.76 0.79 0.79
query15 0.82 0.79 0.79
query16 0.34 0.35 0.34
query17 0.90 0.86 0.91
query18 0.18 0.16 0.16
query19 1.80 1.68 1.67
query20 0.02 0.01 0.01
query21 15.40 0.61 0.62
query22 2.81 3.79 2.81
query23 17.19 0.99 0.93
query24 2.07 0.55 0.57
query25 0.63 0.07 0.06
query26 0.16 0.13 0.15
query27 0.05 0.04 0.04
query28 11.83 0.81 0.81
query29 12.50 3.28 3.41
query30 0.56 0.47 0.44
query31 2.80 0.36 0.37
query32 3.35 0.47 0.48
query33 3.14 3.14 3.18
query34 15.41 4.51 4.49
query35 4.51 4.49 4.48
query36 1.08 0.96 0.96
query37 0.08 0.05 0.05
query38 0.05 0.03 0.03
query39 0.02 0.01 0.01
query40 0.18 0.15 0.15
query41 0.07 0.01 0.02
query42 0.02 0.01 0.01
query43 0.03 0.02 0.02
Total cold run time: 104.45 s
Total hot run time: 31.48 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit e1cac9c6c23948c819b613d94c140b489648bc87 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.3 seconds inserted 10000000 Rows, about 699K ops/s
PR approved by at least one committer and no changes requested.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}