doris
doris copied to clipboard
apache
[Enhancement](group commit)Optimize be select for group commit
Proposed changes
Issue Number: close #xxx
-
Streamload and insert into, if batched and sent to the master FE, should use a consistent BE strategy (previously, insert into reused the first selected BE, while streamload used round robin). First, a map <table id, be id> records a fixed be id for a certain table. The first time a table is imported, a BE is randomly selected, and this table id and be id are recorded in the map permanently. Subsequently, all data imported into this table will select the BE corresponding to the table id recorded in the map. This ensures that batching is maximized to a single BE. To address the issue of excessive load on a single BE, a variable similar to a bvar window is used to monitor the total data volume sent to a specific BE for a specific table during the batch interval (default 10 seconds). A second map <be id, window variable> is used to track this. If a new import finds that its corresponding BE's window variable is less than a certain value (e.g., 1G), the new import continues to be sent to the corresponding BE according to map1. If it exceeds this value, the new import is sent to another BE with the smallest window variable value, and map1 is updated. If every BE exceeds this value, the one with the smallest value is still chosen. This helps to alleviate excessive pressure on a single BE.
-
For streamload, if batched and sent to a BE, it will batch directly on this BE and will commit the transaction at the end of the import. At this point, a request is sent to the FE, which records the size of this import and adds it to the window variable.
-
Streamload sent to observer FE, as well as insert into sent to observer FE, follow the logic in 1 by RPC, passing the table id to the master FE to obtain the selected be id.
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
TPC-H: Total hot run time: 41637 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8cc9df72907d98faba25c5f80301c9d013fdc9e9, data reload: false
------ Round 1 ----------------------------------
q1 17619 4397 4260 4260
q2 2035 209 213 209
q3 10421 1226 1194 1194
q4 10210 886 800 800
q5 7494 2720 2727 2720
q6 214 135 139 135
q7 974 623 609 609
q8 9224 2166 2132 2132
q9 9483 6772 6831 6772
q10 9170 3896 3954 3896
q11 435 255 245 245
q12 419 223 234 223
q13 17340 3221 3282 3221
q14 249 214 222 214
q15 506 479 476 476
q16 493 398 400 398
q17 996 704 693 693
q18 8449 7852 7884 7852
q19 4649 1620 1614 1614
q20 659 332 316 316
q21 5188 3329 3414 3329
q22 382 329 329 329
Total cold run time: 116609 ms
Total hot run time: 41637 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4554 4439 4414 4414
q2 378 266 281 266
q3 3177 2879 2914 2879
q4 2003 1583 1647 1583
q5 5393 5531 5518 5518
q6 214 127 128 127
q7 2179 1785 1872 1785
q8 3239 3451 3396 3396
q9 8625 8731 8693 8693
q10 4108 3722 3803 3722
q11 592 498 502 498
q12 816 630 634 630
q13 16026 3147 3141 3141
q14 303 282 263 263
q15 546 492 480 480
q16 515 444 421 421
q17 1819 1512 1510 1510
q18 7844 7510 7499 7499
q19 1681 1577 1585 1577
q20 1992 1768 1775 1768
q21 4852 4687 4710 4687
q22 574 514 502 502
Total cold run time: 71430 ms
Total hot run time: 55359 ms
TPC-DS: Total hot run time: 172013 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8cc9df72907d98faba25c5f80301c9d013fdc9e9, data reload: false
query1 929 375 368 368
query2 6445 2430 2287 2287
query3 6659 212 211 211
query4 20707 17282 17188 17188
query5 4101 421 414 414
query6 242 153 155 153
query7 4581 296 285 285
query8 301 265 272 265
query9 8581 2421 2393 2393
query10 453 288 262 262
query11 10560 10170 10089 10089
query12 155 87 88 87
query13 1636 359 366 359
query14 9465 8302 7477 7477
query15 235 190 191 190
query16 7444 265 257 257
query17 1309 517 514 514
query18 1890 273 271 271
query19 194 148 147 147
query20 94 82 83 82
query21 190 124 128 124
query22 4259 4037 3847 3847
query23 33453 33042 33327 33042
query24 5037 2803 2846 2803
query25 470 349 364 349
query26 701 152 149 149
query27 1845 321 331 321
query28 3705 2097 2087 2087
query29 839 616 607 607
query30 225 150 154 150
query31 917 762 738 738
query32 59 55 54 54
query33 432 270 258 258
query34 852 479 476 476
query35 698 605 593 593
query36 1039 898 920 898
query37 107 67 72 67
query38 2854 2796 2781 2781
query39 846 804 793 793
query40 200 127 122 122
query41 52 51 50 50
query42 101 95 95 95
query43 578 525 524 524
query44 1063 731 755 731
query45 184 170 170 170
query46 1049 704 760 704
query47 1824 1782 1759 1759
query48 367 291 294 291
query49 764 365 384 365
query50 760 385 374 374
query51 6900 6693 6697 6693
query52 108 90 93 90
query53 352 283 292 283
query54 521 431 417 417
query55 77 72 75 72
query56 266 274 248 248
query57 1119 1058 1036 1036
query58 219 211 210 210
query59 3442 3095 3220 3095
query60 268 259 259 259
query61 88 84 83 83
query62 558 461 443 443
query63 317 284 286 284
query64 2632 1740 1686 1686
query65 3172 3079 3068 3068
query66 698 324 319 319
query67 15242 15002 14787 14787
query68 4514 543 538 538
query69 429 271 270 270
query70 1202 1077 1145 1077
query71 403 279 274 274
query72 7446 5204 5441 5204
query73 717 317 318 317
query74 6018 5580 5651 5580
query75 3292 2632 2618 2618
query76 2112 991 961 961
query77 373 270 268 268
query78 10300 9797 9752 9752
query79 2480 516 511 511
query80 904 425 420 420
query81 510 228 222 222
query82 806 101 88 88
query83 249 169 169 169
query84 247 85 84 84
query85 947 266 261 261
query86 453 337 305 305
query87 3307 3071 3165 3071
query88 4235 2332 2331 2331
query89 492 405 387 387
query90 2029 180 188 180
query91 123 97 93 93
query92 66 50 49 49
query93 1943 506 496 496
query94 1146 179 179 179
query95 402 301 306 301
query96 578 272 261 261
query97 3218 3052 3002 3002
query98 248 219 208 208
query99 1175 851 842 842
Total cold run time: 249794 ms
Total hot run time: 172013 ms
ClickBench: Total hot run time: 30.38 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8cc9df72907d98faba25c5f80301c9d013fdc9e9, data reload: false
query1 0.03 0.03 0.04
query2 0.08 0.04 0.04
query3 0.22 0.05 0.05
query4 1.69 0.06 0.06
query5 0.52 0.48 0.50
query6 1.12 0.71 0.71
query7 0.02 0.02 0.01
query8 0.04 0.04 0.04
query9 0.53 0.47 0.50
query10 0.54 0.54 0.53
query11 0.16 0.11 0.11
query12 0.14 0.12 0.11
query13 0.60 0.59 0.59
query14 0.78 0.76 0.76
query15 0.84 0.81 0.80
query16 0.37 0.37 0.37
query17 1.02 1.02 0.96
query18 0.23 0.25 0.27
query19 1.88 1.67 1.78
query20 0.01 0.01 0.01
query21 15.50 0.69 0.68
query22 4.95 6.53 1.77
query23 18.24 1.36 1.26
query24 1.95 0.22 0.21
query25 0.16 0.09 0.09
query26 0.26 0.17 0.16
query27 0.08 0.08 0.08
query28 13.31 1.01 0.99
query29 13.27 3.42 3.35
query30 0.24 0.05 0.06
query31 2.87 0.38 0.38
query32 3.30 0.47 0.47
query33 2.86 2.93 2.85
query34 17.23 4.54 4.41
query35 4.54 4.54 4.62
query36 0.65 0.46 0.45
query37 0.18 0.15 0.15
query38 0.16 0.14 0.14
query39 0.05 0.04 0.03
query40 0.16 0.14 0.15
query41 0.08 0.04 0.04
query42 0.05 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 110.95 s
Total hot run time: 30.38 s
TeamCity be ut coverage result: Function Coverage: 35.89% (9122/25417) Line Coverage: 27.36% (74954/273905) Region Coverage: 26.60% (38805/145874) Branch Coverage: 23.44% (19749/84240) Coverage Report: http://coverage.selectdb-in.cc/coverage/8cc9df72907d98faba25c5f80301c9d013fdc9e9_8cc9df72907d98faba25c5f80301c9d013fdc9e9/report/index.html
TPC-H: Total hot run time: 42133 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 69a7be4f6a203de575602ef22a242f8f2a310176, data reload: false
------ Round 1 ----------------------------------
q1 17602 4495 4252 4252
q2 2020 195 193 193
q3 10467 1166 1239 1166
q4 10191 811 906 811
q5 7472 2750 2755 2750
q6 218 134 134 134
q7 985 622 592 592
q8 9244 2161 2123 2123
q9 9692 6717 6757 6717
q10 9175 3971 3929 3929
q11 464 254 240 240
q12 433 230 241 230
q13 18422 3212 3232 3212
q14 260 217 211 211
q15 503 466 471 466
q16 497 388 401 388
q17 1001 656 799 656
q18 8468 7920 7772 7772
q19 4234 1620 1590 1590
q20 659 316 315 315
q21 5166 4057 4096 4057
q22 410 329 333 329
Total cold run time: 117583 ms
Total hot run time: 42133 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4585 4509 4393 4393
q2 392 279 270 270
q3 3164 2963 2860 2860
q4 1937 1639 1612 1612
q5 5522 5490 5542 5490
q6 217 125 121 121
q7 2174 1813 1794 1794
q8 3289 3508 3391 3391
q9 8704 8795 8727 8727
q10 3995 3735 3874 3735
q11 610 525 514 514
q12 832 622 629 622
q13 16133 3179 3203 3179
q14 307 278 271 271
q15 542 488 471 471
q16 483 439 448 439
q17 1821 1523 1470 1470
q18 7654 7668 7623 7623
q19 1684 1522 1594 1522
q20 2050 1811 1805 1805
q21 10878 4720 4673 4673
q22 589 544 541 541
Total cold run time: 77562 ms
Total hot run time: 55523 ms
TeamCity be ut coverage result: Function Coverage: 35.99% (9148/25419) Line Coverage: 27.43% (75143/273912) Region Coverage: 26.64% (38891/145961) Branch Coverage: 23.49% (19801/84310) Coverage Report: http://coverage.selectdb-in.cc/coverage/69a7be4f6a203de575602ef22a242f8f2a310176_69a7be4f6a203de575602ef22a242f8f2a310176/report/index.html
TPC-DS: Total hot run time: 168549 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 69a7be4f6a203de575602ef22a242f8f2a310176, data reload: false
query1 926 375 369 369
query2 7563 2414 2388 2388
query3 6642 225 214 214
query4 19240 17321 16909 16909
query5 4113 418 427 418
query6 246 164 169 164
query7 4589 304 285 285
query8 303 277 277 277
query9 8657 2400 2384 2384
query10 444 281 260 260
query11 10513 10032 10002 10002
query12 139 90 88 88
query13 1663 352 351 351
query14 8623 8333 6840 6840
query15 226 196 198 196
query16 7727 262 258 258
query17 1549 520 513 513
query18 1943 273 269 269
query19 202 152 148 148
query20 97 86 103 86
query21 202 139 135 135
query22 4118 3976 3914 3914
query23 33462 32826 33117 32826
query24 5689 2852 2844 2844
query25 465 368 351 351
query26 693 156 151 151
query27 1847 316 312 312
query28 3819 2061 2059 2059
query29 842 594 582 582
query30 227 147 150 147
query31 929 754 766 754
query32 58 52 52 52
query33 493 272 277 272
query34 886 475 485 475
query35 726 585 593 585
query36 1051 932 912 912
query37 102 65 64 64
query38 2874 2759 2747 2747
query39 858 786 782 782
query40 196 128 120 120
query41 48 44 45 44
query42 103 97 97 97
query43 581 551 535 535
query44 1107 724 733 724
query45 180 162 164 162
query46 1054 714 710 710
query47 1835 1765 1785 1765
query48 365 290 287 287
query49 764 373 391 373
query50 766 376 380 376
query51 6919 6825 6870 6825
query52 100 94 88 88
query53 361 288 287 287
query54 558 421 417 417
query55 73 73 71 71
query56 256 237 242 237
query57 1099 1071 1060 1060
query58 234 226 222 222
query59 3379 3394 3103 3103
query60 277 254 251 251
query61 93 89 89 89
query62 533 470 446 446
query63 308 287 287 287
query64 2658 1720 1726 1720
query65 3165 3073 3123 3073
query66 791 336 332 332
query67 15211 14738 14927 14738
query68 4560 532 522 522
query69 442 268 266 266
query70 1203 1155 1127 1127
query71 377 276 309 276
query72 7525 4168 2575 2575
query73 710 321 318 318
query74 5990 5523 5600 5523
query75 3458 2583 2614 2583
query76 2277 1009 1017 1009
query77 405 263 267 263
query78 10315 9692 9830 9692
query79 2453 520 517 517
query80 1090 427 423 423
query81 516 227 222 222
query82 708 92 90 90
query83 247 165 169 165
query84 250 89 82 82
query85 1422 305 311 305
query86 496 317 317 317
query87 3287 3144 3136 3136
query88 4043 2350 2345 2345
query89 475 399 403 399
query90 2064 191 190 190
query91 204 94 101 94
query92 58 48 48 48
query93 2323 501 487 487
query94 1179 179 185 179
query95 399 309 303 303
query96 594 266 265 265
query97 3137 2954 3043 2954
query98 248 211 214 211
query99 1224 841 843 841
Total cold run time: 251439 ms
Total hot run time: 168549 ms
ClickBench: Total hot run time: 30.77 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 69a7be4f6a203de575602ef22a242f8f2a310176, data reload: false
query1 0.05 0.03 0.04
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.68 0.06 0.06
query5 0.50 0.49 0.51
query6 1.12 0.72 0.72
query7 0.02 0.02 0.02
query8 0.05 0.04 0.05
query9 0.55 0.48 0.49
query10 0.54 0.56 0.55
query11 0.16 0.11 0.11
query12 0.14 0.12 0.13
query13 0.60 0.60 0.59
query14 0.77 0.79 0.79
query15 0.82 0.81 0.82
query16 0.35 0.37 0.39
query17 0.95 1.00 0.97
query18 0.22 0.25 0.22
query19 1.82 1.73 1.68
query20 0.02 0.01 0.01
query21 15.74 0.66 0.66
query22 4.31 6.65 2.11
query23 18.29 1.32 1.25
query24 1.30 0.42 0.22
query25 0.15 0.08 0.08
query26 0.26 0.17 0.16
query27 0.07 0.07 0.08
query28 13.42 1.01 0.99
query29 13.74 3.32 3.32
query30 0.24 0.06 0.06
query31 2.89 0.39 0.38
query32 3.25 0.47 0.47
query33 2.92 2.89 2.93
query34 16.99 4.44 4.46
query35 4.66 4.48 4.60
query36 0.66 0.46 0.49
query37 0.18 0.15 0.16
query38 0.15 0.15 0.14
query39 0.04 0.04 0.03
query40 0.18 0.14 0.14
query41 0.09 0.05 0.05
query42 0.06 0.04 0.05
query43 0.04 0.03 0.03
Total cold run time: 110.3 s
Total hot run time: 30.77 s
TPC-H: Total hot run time: 41643 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 201dff1a9bf88c54d3e85a3528e0d014a0749bae, data reload: false
------ Round 1 ----------------------------------
q1 17614 4481 4276 4276
q2 2032 200 204 200
q3 10430 1248 1116 1116
q4 10187 813 862 813
q5 7486 2722 2758 2722
q6 223 139 137 137
q7 964 627 631 627
q8 9225 2194 2109 2109
q9 9389 6741 6738 6738
q10 9678 3986 3856 3856
q11 443 241 252 241
q12 444 250 251 250
q13 17261 3181 3230 3181
q14 275 237 228 228
q15 548 478 470 470
q16 470 376 378 376
q17 980 607 630 607
q18 8338 7831 7752 7752
q19 6173 1418 1369 1369
q20 671 348 314 314
q21 5159 3990 3928 3928
q22 403 339 333 333
Total cold run time: 118393 ms
Total hot run time: 41643 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4629 4416 4446 4416
q2 365 267 269 267
q3 3180 3005 3035 3005
q4 1935 1618 1682 1618
q5 5507 5555 5506 5506
q6 221 133 136 133
q7 2192 1814 1810 1810
q8 3262 3405 3390 3390
q9 8661 8686 8773 8686
q10 4074 3876 3685 3685
q11 582 481 494 481
q12 806 615 620 615
q13 17218 3125 3133 3125
q14 304 281 262 262
q15 529 483 478 478
q16 498 426 437 426
q17 1828 1520 1500 1500
q18 7968 7359 7264 7264
q19 1696 1607 1478 1478
q20 2090 1764 1780 1764
q21 4971 4715 4812 4715
q22 673 542 555 542
Total cold run time: 73189 ms
Total hot run time: 55166 ms
TPC-DS: Total hot run time: 172238 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 201dff1a9bf88c54d3e85a3528e0d014a0749bae, data reload: false
query1 939 382 385 382
query2 6459 2429 2406 2406
query3 6644 206 207 206
query4 19216 17272 17319 17272
query5 4171 457 454 454
query6 263 159 152 152
query7 4589 303 301 301
query8 308 275 287 275
query9 8552 2454 2430 2430
query10 445 288 270 270
query11 10730 10112 10037 10037
query12 130 87 91 87
query13 1638 361 350 350
query14 9331 7552 6613 6613
query15 230 191 190 190
query16 7824 271 267 267
query17 1680 528 524 524
query18 1934 279 279 279
query19 205 152 155 152
query20 97 83 108 83
query21 211 129 126 126
query22 4333 4069 3963 3963
query23 33583 33000 33263 33000
query24 11133 2762 2748 2748
query25 638 377 356 356
query26 1395 153 151 151
query27 3026 323 322 322
query28 7431 2076 2055 2055
query29 941 612 610 610
query30 272 148 155 148
query31 958 755 726 726
query32 92 52 54 52
query33 764 283 285 283
query34 969 488 473 473
query35 765 622 620 620
query36 1099 922 930 922
query37 159 99 76 76
query38 2877 2763 2768 2763
query39 857 779 791 779
query40 213 126 121 121
query41 55 53 53 53
query42 122 98 104 98
query43 592 553 552 552
query44 1223 730 739 730
query45 199 164 164 164
query46 1081 758 713 713
query47 1867 1783 1805 1783
query48 382 294 299 294
query49 1081 402 408 402
query50 786 388 393 388
query51 6715 6669 6641 6641
query52 98 97 92 92
query53 358 289 295 289
query54 844 458 442 442
query55 74 75 76 75
query56 307 255 260 255
query57 1135 1050 1069 1050
query58 254 253 242 242
query59 3423 3151 3169 3151
query60 292 278 266 266
query61 90 88 93 88
query62 660 453 435 435
query63 324 284 294 284
query64 8911 2215 1749 1749
query65 3188 3080 3128 3080
query66 925 344 326 326
query67 15364 15087 14875 14875
query68 4522 543 530 530
query69 467 315 302 302
query70 1154 1078 1101 1078
query71 414 271 267 267
query72 7091 5626 5503 5503
query73 744 332 322 322
query74 5988 5482 5509 5482
query75 3637 2670 2684 2670
query76 2878 948 923 923
query77 465 291 285 285
query78 10243 9688 9749 9688
query79 2336 516 513 513
query80 1098 466 467 466
query81 590 223 215 215
query82 1013 102 100 100
query83 237 165 162 162
query84 247 85 91 85
query85 1180 281 280 280
query86 457 330 302 302
query87 3283 3056 3085 3056
query88 3571 2463 2461 2461
query89 483 403 376 376
query90 1793 196 255 196
query91 124 97 102 97
query92 73 53 48 48
query93 1683 515 491 491
query94 1214 194 189 189
query95 401 314 316 314
query96 585 280 266 266
query97 3191 3034 2992 2992
query98 222 198 203 198
query99 1205 848 840 840
Total cold run time: 270040 ms
Total hot run time: 172238 ms
ClickBench: Total hot run time: 30.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 201dff1a9bf88c54d3e85a3528e0d014a0749bae, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.67 0.08 0.08
query5 0.50 0.49 0.48
query6 1.13 0.72 0.72
query7 0.02 0.02 0.02
query8 0.05 0.04 0.05
query9 0.54 0.48 0.49
query10 0.54 0.54 0.53
query11 0.16 0.11 0.11
query12 0.14 0.11 0.12
query13 0.59 0.58 0.59
query14 0.79 0.77 0.78
query15 0.83 0.81 0.82
query16 0.36 0.35 0.36
query17 0.97 0.96 0.98
query18 0.22 0.26 0.25
query19 1.90 1.77 1.73
query20 0.02 0.01 0.01
query21 15.49 0.68 0.66
query22 4.67 6.42 2.16
query23 18.19 1.42 1.32
query24 2.16 0.22 0.23
query25 0.17 0.08 0.08
query26 0.25 0.17 0.18
query27 0.07 0.09 0.08
query28 13.23 1.02 0.99
query29 12.72 3.44 3.35
query30 0.25 0.06 0.05
query31 2.87 0.37 0.38
query32 3.26 0.46 0.48
query33 2.84 2.87 2.92
query34 17.03 4.44 4.43
query35 4.49 4.54 4.53
query36 0.71 0.49 0.49
query37 0.18 0.14 0.15
query38 0.15 0.14 0.14
query39 0.05 0.03 0.03
query40 0.18 0.13 0.15
query41 0.09 0.05 0.05
query42 0.06 0.05 0.04
query43 0.04 0.04 0.03
Total cold run time: 109.93 s
Total hot run time: 30.94 s
TeamCity be ut coverage result: Function Coverage: 36.45% (8973/24615) Line Coverage: 27.98% (73317/262074) Region Coverage: 27.38% (38010/138818) Branch Coverage: 24.00% (19299/80422) Coverage Report: http://coverage.selectdb-in.cc/coverage/201dff1a9bf88c54d3e85a3528e0d014a0749bae_201dff1a9bf88c54d3e85a3528e0d014a0749bae/report/index.html