doris
doris copied to clipboard
Published
20 hours ago •
apache
apache
[improve](cloud) make meta_service_endpoint configurable
Proposed changes
When scale out the meta service, we need to change the meta_service_endpoint conf item in both fe and be, and restart all the service. This is not the best practice in a cloud environment. Using admin set all frontends config to change the config and without restarting it is good practice.
In order to prevent setting a wrong meta_service_endpoint config, it has a constraint: new config and old config should have endpoint intersect to indicate they belong to the same cluster
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.
Please clearly describe your PR:
- What problem was fixed (it's best to include specific error reporting information). How it was fixed.
- Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
- What features were added. Why was this function added?
- Which code was refactored and why was this part of the code refactored?
- Which functions were optimized and what is the difference before and after the optimization?
TPC-H: Total hot run time: 33942 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f595575809916df99b7ce6ffc3cd61916fb4019b, data reload: false
------ Round 1 ----------------------------------
q1 17612 5158 5042 5042
q2 1923 279 183 183
q3 10310 1345 720 720
q4 10228 1033 516 516
q5 7537 2350 2436 2350
q6 178 169 134 134
q7 902 753 603 603
q8 9336 1322 1124 1124
q9 7038 5181 5111 5111
q10 6884 2296 1890 1890
q11 487 288 266 266
q12 340 356 210 210
q13 17803 3676 3077 3077
q14 218 244 213 213
q15 564 480 489 480
q16 423 425 380 380
q17 604 873 354 354
q18 7722 7216 7182 7182
q19 1210 959 552 552
q20 335 337 226 226
q21 3855 2566 2361 2361
q22 1018 1033 968 968
Total cold run time: 106527 ms
Total hot run time: 33942 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5147 5067 5081 5067
q2 232 309 216 216
q3 2162 2635 2340 2340
q4 1328 1786 1343 1343
q5 4218 4108 4163 4108
q6 208 167 124 124
q7 1900 1827 1628 1628
q8 2515 2520 2498 2498
q9 6879 6770 6857 6770
q10 2889 3086 2666 2666
q11 555 494 467 467
q12 649 720 608 608
q13 3326 3710 3074 3074
q14 280 283 272 272
q15 499 471 462 462
q16 421 465 427 427
q17 1113 1512 1362 1362
q18 7295 7172 7226 7172
q19 789 844 1002 844
q20 1903 1976 1840 1840
q21 4868 4383 4328 4328
q22 1078 1023 975 975
Total cold run time: 50254 ms
Total hot run time: 48591 ms
TPC-DS: Total hot run time: 185415 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f595575809916df99b7ce6ffc3cd61916fb4019b, data reload: false
query1 1007 384 390 384
query2 2613 1810 1805 1805
query3 6637 222 229 222
query4 25845 23724 23294 23294
query5 3258 632 461 461
query6 296 205 197 197
query7 4611 491 284 284
query8 262 235 217 217
query9 8488 2620 2618 2618
query10 500 355 277 277
query11 15708 15058 14743 14743
query12 154 109 105 105
query13 1649 540 426 426
query14 9193 6097 6085 6085
query15 201 196 165 165
query16 5350 627 501 501
query17 1422 748 593 593
query18 1681 416 315 315
query19 259 195 175 175
query20 124 125 125 125
query21 218 124 109 109
query22 4310 4403 4206 4206
query23 34432 32924 33073 32924
query24 8084 2363 2350 2350
query25 491 459 401 401
query26 1220 274 151 151
query27 2704 516 361 361
query28 4295 2157 2127 2127
query29 740 554 426 426
query30 275 218 182 182
query31 875 850 783 783
query32 70 84 63 63
query33 513 361 339 339
query34 784 872 529 529
query35 761 825 722 722
query36 933 985 873 873
query37 115 105 81 81
query38 3993 4150 4032 4032
query39 1446 1450 1402 1402
query40 212 125 107 107
query41 65 59 59 59
query42 124 106 109 106
query43 505 497 481 481
query44 1303 833 839 833
query45 179 182 174 174
query46 848 1028 636 636
query47 1760 1805 1754 1754
query48 380 427 325 325
query49 702 515 410 410
query50 642 693 410 410
query51 4103 4159 4089 4089
query52 109 110 104 104
query53 222 247 188 188
query54 588 570 508 508
query55 83 83 82 82
query56 341 308 293 293
query57 1169 1198 1126 1126
query58 258 261 265 261
query59 2554 2636 2564 2564
query60 350 317 300 300
query61 120 121 121 121
query62 782 716 653 653
query63 221 188 190 188
query64 3055 1019 688 688
query65 4251 4138 4183 4138
query66 904 473 333 333
query67 15809 15552 15320 15320
query68 8284 891 526 526
query69 485 304 270 270
query70 1224 1076 1077 1076
query71 561 314 289 289
query72 5503 4691 4774 4691
query73 692 617 357 357
query74 8821 9037 8581 8581
query75 4105 3183 2717 2717
query76 3884 1192 800 800
query77 793 383 281 281
query78 10052 10276 9377 9377
query79 2383 818 596 596
query80 635 537 445 445
query81 484 258 225 225
query82 423 133 97 97
query83 274 245 235 235
query84 289 107 90 90
query85 766 346 318 318
query86 336 302 293 293
query87 4295 4372 4344 4344
query88 3262 2398 2281 2281
query89 384 321 282 282
query90 1945 206 207 206
query91 140 161 110 110
query92 74 65 56 56
query93 1595 933 597 597
query94 662 416 307 307
query95 356 299 283 283
query96 488 571 281 281
query97 2696 2776 2644 2644
query98 230 209 196 196
query99 1424 1390 1260 1260
Total cold run time: 264701 ms
Total hot run time: 185415 ms
ClickBench: Total hot run time: 29.15 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f595575809916df99b7ce6ffc3cd61916fb4019b, data reload: false
query1 0.03 0.04 0.03
query2 0.07 0.03 0.04
query3 0.23 0.08 0.06
query4 1.61 0.10 0.11
query5 0.43 0.41 0.42
query6 1.18 0.67 0.67
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.57 0.53 0.51
query10 0.56 0.57 0.58
query11 0.16 0.10 0.10
query12 0.14 0.12 0.11
query13 0.61 0.61 0.60
query14 0.78 0.81 0.79
query15 0.89 0.85 0.87
query16 0.38 0.38 0.38
query17 1.07 1.02 1.08
query18 0.23 0.21 0.21
query19 1.92 1.83 1.83
query20 0.01 0.01 0.02
query21 15.40 0.93 0.54
query22 0.74 1.14 0.61
query23 15.05 1.34 0.64
query24 6.72 1.19 1.82
query25 0.50 0.22 0.06
query26 0.59 0.17 0.13
query27 0.05 0.05 0.05
query28 9.98 0.86 0.43
query29 12.54 3.95 3.35
query30 0.26 0.09 0.06
query31 2.82 0.58 0.40
query32 3.23 0.55 0.47
query33 3.11 3.02 3.14
query34 15.87 5.10 4.56
query35 4.53 4.56 4.50
query36 0.66 0.49 0.48
query37 0.08 0.06 0.07
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.18 0.13 0.13
query41 0.08 0.02 0.02
query42 0.03 0.03 0.02
query43 0.03 0.03 0.03
Total cold run time: 103.47 s
Total hot run time: 29.15 s
BE UT Coverage Report
Increment line coverage 0.00% (0/1) :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 56.40% (15074/26725) |
| Line Coverage | 45.17% (134784/298404) |
| Region Coverage | 44.28% (67831/153179) |
| Branch Coverage | 38.85% (34812/89604) |
BE Regression && UT Coverage Report
Increment line coverage 0.00% (0/25) :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 79.77% (20980/26302) |
| Line Coverage | 72.70% (216842/298264) |
| Region Coverage | 70.94% (127797/180159) |
| Branch Coverage | 64.58% (66145/102418) |
PR approved by at least one committer and no changes requested.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
TPC-H: Total hot run time: 33939 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ab183fb05417b69ab2c6fe2eb9cd75bc10f538c0, data reload: false
------ Round 1 ----------------------------------
q1 17741 5213 5050 5050
q2 1950 308 187 187
q3 10392 1363 707 707
q4 10258 993 537 537
q5 7632 2421 2348 2348
q6 179 161 132 132
q7 910 742 609 609
q8 9310 1377 1106 1106
q9 6828 5108 5092 5092
q10 6897 2405 1967 1967
q11 485 277 271 271
q12 344 343 214 214
q13 17791 3694 3113 3113
q14 228 220 216 216
q15 572 473 476 473
q16 429 437 370 370
q17 608 865 365 365
q18 7657 7091 7079 7079
q19 2305 970 553 553
q20 332 333 230 230
q21 3856 3167 2309 2309
q22 1074 1012 1011 1011
Total cold run time: 107778 ms
Total hot run time: 33939 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5288 5441 5112 5112
q2 259 327 225 225
q3 2156 2654 2258 2258
q4 1344 1773 1356 1356
q5 4242 4220 4479 4220
q6 220 174 128 128
q7 1957 1889 1791 1791
q8 2585 2489 2508 2489
q9 7291 7208 7110 7110
q10 3117 3235 2801 2801
q11 593 523 491 491
q12 676 781 608 608
q13 3460 3900 3276 3276
q14 288 307 275 275
q15 515 482 489 482
q16 447 479 457 457
q17 1147 1572 1377 1377
q18 7622 7149 7079 7079
q19 787 791 831 791
q20 1960 1945 1795 1795
q21 4717 4327 4269 4269
q22 1112 1047 1024 1024
Total cold run time: 51783 ms
Total hot run time: 49414 ms
TPC-DS: Total hot run time: 186210 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ab183fb05417b69ab2c6fe2eb9cd75bc10f538c0, data reload: false
query1 998 391 390 390
query2 6548 1870 1864 1864
query3 6744 227 226 226
query4 26568 23542 23351 23351
query5 4312 628 475 475
query6 310 214 204 204
query7 4623 511 283 283
query8 269 236 234 234
query9 8615 2627 2636 2627
query10 480 368 305 305
query11 15187 15033 14974 14974
query12 160 109 110 109
query13 1660 524 406 406
query14 9177 6055 6143 6055
query15 185 186 174 174
query16 7339 627 482 482
query17 1206 699 564 564
query18 1967 395 296 296
query19 194 188 160 160
query20 125 130 137 130
query21 214 131 113 113
query22 3905 4207 3946 3946
query23 33956 33144 33175 33144
query24 8447 2367 2372 2367
query25 507 442 388 388
query26 822 263 155 155
query27 2750 507 352 352
query28 4309 2113 2099 2099
query29 704 580 426 426
query30 288 216 193 193
query31 924 846 781 781
query32 75 63 63 63
query33 546 361 322 322
query34 795 857 517 517
query35 783 861 730 730
query36 965 986 887 887
query37 116 99 77 77
query38 4085 4142 4081 4081
query39 1485 1421 1410 1410
query40 220 131 115 115
query41 73 64 64 64
query42 128 115 114 114
query43 513 514 474 474
query44 1305 832 820 820
query45 191 175 172 172
query46 837 1028 642 642
query47 1772 1769 1730 1730
query48 399 430 323 323
query49 754 490 396 396
query50 635 673 412 412
query51 4191 4143 4119 4119
query52 110 107 99 99
query53 228 251 187 187
query54 581 568 496 496
query55 83 87 86 86
query56 313 313 271 271
query57 1179 1192 1142 1142
query58 271 258 256 256
query59 2696 2726 2702 2702
query60 330 317 314 314
query61 127 126 137 126
query62 854 732 645 645
query63 220 183 186 183
query64 3478 1029 666 666
query65 4266 4185 4191 4185
query66 1007 419 314 314
query67 15638 15399 15351 15351
query68 7650 875 511 511
query69 482 305 270 270
query70 1226 1136 1076 1076
query71 421 328 318 318
query72 5793 4829 4859 4829
query73 676 631 349 349
query74 8915 9122 8692 8692
query75 3445 3213 2699 2699
query76 3346 1191 769 769
query77 736 372 290 290
query78 9967 10224 9442 9442
query79 2047 813 591 591
query80 664 511 448 448
query81 490 264 227 227
query82 178 131 98 98
query83 251 255 237 237
query84 253 108 84 84
query85 761 358 356 356
query86 366 291 282 282
query87 4429 4500 4330 4330
query88 3388 2292 2288 2288
query89 377 315 306 306
query90 1924 211 205 205
query91 141 141 116 116
query92 75 67 61 61
query93 2121 927 580 580
query94 658 424 310 310
query95 382 296 282 282
query96 492 583 278 278
query97 2753 2765 2649 2649
query98 247 210 200 200
query99 1304 1427 1273 1273
Total cold run time: 270700 ms
Total hot run time: 186210 ms
ClickBench: Total hot run time: 30.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ab183fb05417b69ab2c6fe2eb9cd75bc10f538c0, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.03 0.04
query3 0.23 0.08 0.08
query4 1.61 0.10 0.10
query5 0.44 0.43 0.42
query6 1.17 0.66 0.68
query7 0.02 0.02 0.02
query8 0.05 0.03 0.04
query9 0.60 0.50 0.51
query10 0.58 0.57 0.57
query11 0.15 0.12 0.11
query12 0.14 0.12 0.11
query13 0.64 0.62 0.60
query14 0.82 0.82 0.83
query15 0.91 0.88 0.88
query16 0.39 0.38 0.40
query17 1.08 1.08 1.07
query18 0.23 0.21 0.21
query19 1.94 1.84 1.84
query20 0.01 0.02 0.01
query21 15.40 0.88 0.53
query22 0.77 1.19 0.69
query23 14.90 1.38 0.66
query24 7.16 1.17 1.24
query25 0.48 0.16 0.07
query26 0.53 0.18 0.15
query27 0.06 0.06 0.05
query28 9.88 0.95 0.47
query29 12.62 3.98 3.37
query30 0.25 0.09 0.06
query31 2.83 0.61 0.40
query32 3.23 0.55 0.47
query33 3.00 3.09 3.15
query34 16.19 5.42 4.78
query35 4.90 4.85 4.83
query36 0.71 0.52 0.49
query37 0.09 0.06 0.06
query38 0.06 0.04 0.04
query39 0.04 0.02 0.02
query40 0.17 0.15 0.13
query41 0.09 0.03 0.02
query42 0.04 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 104.55 s
Total hot run time: 30.07 s
BE UT Coverage Report
Increment line coverage 0.00% (0/1) :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 56.36% (15055/26711) |
| Line Coverage | 45.14% (134710/298411) |
| Region Coverage | 44.28% (67736/152983) |
| Branch Coverage | 38.87% (34762/89440) |
BE Regression && UT Coverage Report
Increment line coverage 0.00% (0/25) :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 61.12% (16066/26285) |
| Line Coverage | 50.68% (151164/298267) |
| Region Coverage | 47.99% (86362/179976) |
| Branch Coverage | 41.55% (42493/102260) |
PR approved by at least one committer and no changes requested.