doris
doris copied to clipboard
apache
[Feature](Streaming Job) Extend streaming job to support MySQL synchronization
What problem does this PR solve?
Issue Number: close #58896
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
-
Test
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
-
Behavior changed:
- [ ] No.
- [ ] Yes.
-
Does this need documentation?
- [ ] No.
- [ ] Yes.
Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.
Please clearly describe your PR:
- What problem was fixed (it's best to include specific error reporting information). How it was fixed.
- Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
- What features were added. Why was this function added?
- Which code was refactored and why was this part of the code refactored?
- Which functions were optimized and what is the difference before and after the optimization?
Cloud UT Coverage Report
Increment line coverage :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 79.82% (1728/2165) |
| Line Coverage | 65.92% (30578/46390) |
| Region Coverage | 66.58% (15247/22900) |
| Branch Coverage | 56.95% (8116/14250) |
FE UT Coverage Report
Increment line coverage 1.32% (13/984) :tada:
Increment coverage report
Complete coverage report
TPC-H: Total hot run time: 35374 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f4e1a9eceb169b2b2201482d22410bc26c1e1db9, data reload: false
------ Round 1 ----------------------------------
q1 17687 4424 4080 4080
q2 2029 361 231 231
q3 10196 1334 761 761
q4 10217 884 313 313
q5 7556 2177 1898 1898
q6 184 175 139 139
q7 1025 891 717 717
q8 9380 1403 1295 1295
q9 7069 5325 5353 5325
q10 6785 2386 1971 1971
q11 540 315 294 294
q12 666 747 571 571
q13 17777 3753 3059 3059
q14 290 298 278 278
q15 580 517 510 510
q16 959 954 870 870
q17 698 865 551 551
q18 7355 7090 6945 6945
q19 907 971 595 595
q20 425 372 242 242
q21 4213 3947 3754 3754
q22 1056 993 975 975
Total cold run time: 107594 ms
Total hot run time: 35374 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4120 4067 4058 4058
q2 326 387 321 321
q3 2177 2680 2309 2309
q4 1334 1774 1313 1313
q5 4254 4756 4767 4756
q6 236 176 131 131
q7 2079 1989 1860 1860
q8 2752 2541 2630 2541
q9 7753 7626 7521 7521
q10 3044 3477 2822 2822
q11 611 511 495 495
q12 634 714 589 589
q13 3706 4022 3454 3454
q14 283 301 282 282
q15 543 513 504 504
q16 905 911 877 877
q17 1250 1484 1446 1446
q18 7951 7565 7505 7505
q19 856 880 868 868
q20 2046 2040 1974 1974
q21 5048 4772 4539 4539
q22 1158 1052 971 971
Total cold run time: 53066 ms
Total hot run time: 51136 ms
TPC-DS: Total hot run time: 181298 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f4e1a9eceb169b2b2201482d22410bc26c1e1db9, data reload: false
query5 5498 648 495 495
query6 330 235 229 229
query7 4219 479 286 286
query8 306 253 243 243
query9 8776 2577 2587 2577
query10 558 394 332 332
query11 15472 14761 14583 14583
query12 188 120 118 118
query13 1273 514 414 414
query14 6664 3422 3065 3065
query14_1 2960 2936 2939 2936
query15 208 198 184 184
query16 966 480 488 480
query17 1167 720 602 602
query18 2724 450 361 361
query19 234 238 211 211
query20 120 116 112 112
query21 225 147 119 119
query22 3986 4002 3855 3855
query23 16602 16188 15911 15911
query23_1 16001 16017 16055 16017
query24 7292 1679 1259 1259
query24_1 1270 1261 1244 1244
query25 588 497 442 442
query26 1262 265 166 166
query27 2746 483 322 322
query28 4456 2195 2169 2169
query29 832 576 471 471
query30 322 252 223 223
query31 838 703 606 606
query32 88 71 70 70
query33 548 350 303 303
query34 940 926 561 561
query35 790 816 748 748
query36 838 910 827 827
query37 146 93 77 77
query38 3869 3870 3817 3817
query39 757 742 707 707
query39_1 698 705 682 682
query40 228 133 124 124
query41 78 61 61 61
query42 108 107 107 107
query43 424 439 412 412
query44 1344 757 789 757
query45 195 187 183 183
query46 888 979 631 631
query47 1646 1677 1589 1589
query48 318 358 249 249
query49 632 433 359 359
query50 670 313 224 224
query51 3820 3829 3847 3829
query52 110 111 98 98
query53 323 354 294 294
query54 289 264 256 256
query55 78 79 71 71
query56 305 307 297 297
query57 1120 1141 1062 1062
query58 313 260 248 248
query59 2416 2454 2346 2346
query60 316 310 292 292
query61 166 159 157 157
query62 702 645 634 634
query63 329 295 305 295
query64 4906 1297 1007 1007
query65 3987 3955 3984 3955
query66 1363 440 327 327
query67 15221 14928 14831 14831
query68 8524 1037 751 751
query69 492 349 316 316
query70 1099 991 1021 991
query71 399 316 286 286
query72 6039 5013 5240 5013
query73 730 689 318 318
query74 8929 8953 8646 8646
query75 3598 3532 3156 3156
query76 4045 1161 760 760
query77 578 451 299 299
query78 9383 9690 8813 8813
query79 1629 886 620 620
query80 710 654 564 564
query81 535 269 244 244
query82 224 129 106 106
query83 260 250 239 239
query84 261 117 106 106
query85 885 501 454 454
query86 355 273 305 273
query87 4043 4099 3990 3990
query88 4373 2327 2295 2295
query89 466 431 389 389
query90 2206 157 154 154
query91 172 163 136 136
query92 77 69 60 60
query93 1638 934 584 584
query94 463 300 269 269
query95 543 327 361 327
query96 585 482 217 217
query97 2571 2657 2566 2566
query98 224 192 195 192
query99 1323 1284 1211 1211
Total cold run time: 266440 ms
Total hot run time: 181298 ms
Cloud UT Coverage Report
Increment line coverage :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 79.82% (1728/2165) |
| Line Coverage | 65.91% (30577/46390) |
| Region Coverage | 66.60% (15251/22900) |
| Branch Coverage | 56.93% (8112/14250) |
ClickBench: Total hot run time: 27.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f4e1a9eceb169b2b2201482d22410bc26c1e1db9, data reload: false
query1 0.05 0.06 0.05
query2 0.10 0.05 0.04
query3 0.27 0.09 0.08
query4 1.64 0.11 0.11
query5 0.26 0.25 0.26
query6 1.16 0.64 0.64
query7 0.03 0.03 0.02
query8 0.06 0.04 0.04
query9 0.57 0.51 0.52
query10 0.56 0.56 0.56
query11 0.15 0.10 0.11
query12 0.14 0.10 0.11
query13 0.62 0.61 0.60
query14 0.97 0.97 0.98
query15 0.83 0.82 0.82
query16 0.38 0.41 0.40
query17 1.07 1.10 1.01
query18 0.22 0.21 0.21
query19 1.88 1.90 1.79
query20 0.02 0.01 0.01
query21 15.45 0.31 0.13
query22 4.85 0.05 0.05
query23 16.03 0.30 0.10
query24 2.08 0.74 0.74
query25 0.09 0.07 0.05
query26 0.13 0.13 0.14
query27 0.06 0.08 0.05
query28 5.39 1.21 1.02
query29 12.60 4.02 3.30
query30 0.28 0.14 0.11
query31 2.82 0.63 0.39
query32 3.24 0.55 0.46
query33 3.04 3.00 3.00
query34 16.94 5.25 4.56
query35 4.53 4.55 4.60
query36 0.65 0.49 0.51
query37 0.10 0.06 0.06
query38 0.07 0.05 0.05
query39 0.04 0.03 0.03
query40 0.17 0.14 0.14
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 99.71 s
Total hot run time: 27.81 s
Cloud UT Coverage Report
Increment line coverage :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 79.82% (1728/2165) |
| Line Coverage | 65.92% (30581/46390) |
| Region Coverage | 66.62% (15256/22900) |
| Branch Coverage | 56.95% (8115/14250) |
Cloud UT Coverage Report
Increment line coverage :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 79.82% (1728/2165) |
| Line Coverage | 65.93% (30586/46390) |
| Region Coverage | 66.62% (15255/22900) |
| Branch Coverage | 56.91% (8110/14250) |
FE UT Coverage Report
Increment line coverage 1.30% (13/1002) :tada:
Increment coverage report
Complete coverage report
Cloud UT Coverage Report
Increment line coverage :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 79.83% (1730/2167) |
| Line Coverage | 65.89% (30622/46476) |
| Region Coverage | 66.59% (15272/22934) |
| Branch Coverage | 56.91% (8119/14266) |
Cloud UT Coverage Report
Increment line coverage :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 79.83% (1730/2167) |
| Line Coverage | 65.90% (30626/46476) |
| Region Coverage | 66.63% (15280/22934) |
| Branch Coverage | 56.94% (8123/14266) |
FE Regression Coverage Report
Increment line coverage 61.47% (584/950) :tada:
Increment coverage report
Complete coverage report
Cloud UT Coverage Report
Increment line coverage :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 79.83% (1730/2167) |
| Line Coverage | 65.89% (30623/46476) |
| Region Coverage | 66.58% (15270/22934) |
| Branch Coverage | 56.91% (8119/14266) |
TPC-H: Total hot run time: 35213 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2dd280b48f3f20b09fe98cc59748540e1ed9e93c, data reload: false
------ Round 1 ----------------------------------
q1 17715 4308 4092 4092
q2 2009 357 236 236
q3 10456 1341 734 734
q4 10329 849 329 329
q5 9536 2173 1934 1934
q6 213 165 134 134
q7 988 874 726 726
q8 9354 1470 1112 1112
q9 7380 5336 5344 5336
q10 6857 2417 1952 1952
q11 528 317 301 301
q12 721 727 570 570
q13 17802 3722 3001 3001
q14 301 306 273 273
q15 606 533 509 509
q16 708 662 621 621
q17 711 726 633 633
q18 7826 7160 7134 7134
q19 1108 973 612 612
q20 404 358 248 248
q21 4285 3928 3771 3771
q22 1043 1002 955 955
Total cold run time: 110880 ms
Total hot run time: 35213 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4170 4277 4058 4058
q2 330 413 305 305
q3 2156 2670 2299 2299
q4 1352 1737 1297 1297
q5 4646 4464 4819 4464
q6 247 186 134 134
q7 2103 1950 1773 1773
q8 2710 2476 2514 2476
q9 7585 7477 7463 7463
q10 3116 3268 2821 2821
q11 593 543 539 539
q12 910 740 607 607
q13 3480 3924 3409 3409
q14 328 313 282 282
q15 572 534 518 518
q16 637 682 633 633
q17 1204 1487 1490 1487
q18 7903 7700 7652 7652
q19 913 875 932 875
q20 2073 2026 1805 1805
q21 4639 4304 4155 4155
q22 1121 1032 988 988
Total cold run time: 52788 ms
Total hot run time: 50040 ms
TPC-DS: Total hot run time: 178382 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2dd280b48f3f20b09fe98cc59748540e1ed9e93c, data reload: false
query5 4392 644 481 481
query6 326 224 227 224
query7 4218 491 271 271
query8 298 246 233 233
query9 8751 2566 2573 2566
query10 516 383 322 322
query11 15580 14821 14624 14624
query12 173 116 128 116
query13 1269 525 404 404
query14 5784 3346 3023 3023
query14_1 2861 2852 2939 2852
query15 225 205 180 180
query16 925 481 456 456
query17 1151 752 603 603
query18 2479 461 356 356
query19 255 247 216 216
query20 127 117 111 111
query21 223 150 118 118
query22 4009 4200 4043 4043
query23 16808 16291 16269 16269
query23_1 16145 16033 15957 15957
query24 7359 1797 1227 1227
query24_1 1251 1242 1280 1242
query25 590 527 459 459
query26 1268 284 165 165
query27 2736 485 312 312
query28 4460 2168 2173 2168
query29 837 606 524 524
query30 319 244 225 225
query31 832 718 606 606
query32 74 73 70 70
query33 544 348 293 293
query34 926 974 535 535
query35 806 835 715 715
query36 893 914 834 834
query37 128 94 75 75
query38 2829 2822 2831 2822
query39 748 741 731 731
query39_1 689 725 691 691
query40 229 139 120 120
query41 65 62 62 62
query42 113 102 102 102
query43 433 452 404 404
query44 1370 756 744 744
query45 195 193 183 183
query46 947 1050 623 623
query47 1680 1727 1627 1627
query48 318 331 258 258
query49 645 445 354 354
query50 686 299 219 219
query51 3770 3825 3845 3825
query52 111 113 102 102
query53 325 384 292 292
query54 300 264 266 264
query55 84 77 75 75
query56 307 298 295 295
query57 1154 1135 1080 1080
query58 294 256 249 249
query59 2392 2481 2367 2367
query60 312 318 294 294
query61 162 196 154 154
query62 748 699 616 616
query63 345 291 294 291
query64 4998 1313 991 991
query65 4073 3931 3934 3931
query66 1459 505 315 315
query67 15169 14976 14911 14911
query68 8210 1019 734 734
query69 508 352 312 312
query70 1110 969 973 969
query71 369 316 284 284
query72 6162 5002 4952 4952
query73 676 600 314 314
query74 8888 8790 8572 8572
query75 3396 3312 2753 2753
query76 3876 1251 775 775
query77 531 411 284 284
query78 9499 9499 8806 8806
query79 2041 910 608 608
query80 971 678 559 559
query81 547 277 237 237
query82 451 137 104 104
query83 270 262 232 232
query84 261 121 101 101
query85 943 520 463 463
query86 408 301 276 276
query87 3097 3091 2970 2970
query88 4621 2255 2283 2255
query89 501 436 386 386
query90 2144 165 153 153
query91 179 171 148 148
query92 82 70 69 69
query93 2383 947 554 554
query94 540 311 284 284
query95 568 388 305 305
query96 597 497 212 212
query97 2302 2311 2199 2199
query98 247 199 187 187
query99 1331 1338 1211 1211
Total cold run time: 264012 ms
Total hot run time: 178382 ms
ClickBench: Total hot run time: 28.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2dd280b48f3f20b09fe98cc59748540e1ed9e93c, data reload: false
query1 0.05 0.04 0.04
query2 0.14 0.06 0.06
query3 0.33 0.08 0.07
query4 1.61 0.10 0.10
query5 0.27 0.24 0.26
query6 1.18 0.63 0.63
query7 0.03 0.02 0.03
query8 0.07 0.06 0.06
query9 0.58 0.51 0.50
query10 0.55 0.55 0.56
query11 0.26 0.14 0.13
query12 0.26 0.14 0.15
query13 0.63 0.63 0.60
query14 1.02 1.01 1.01
query15 0.87 0.81 0.80
query16 0.39 0.39 0.40
query17 1.00 0.99 0.98
query18 0.23 0.22 0.21
query19 1.87 1.72 1.86
query20 0.02 0.02 0.02
query21 15.39 0.29 0.24
query22 4.97 0.09 0.10
query23 15.43 0.37 0.22
query24 2.42 0.44 0.28
query25 0.09 0.08 0.10
query26 0.19 0.18 0.17
query27 0.10 0.09 0.09
query28 3.63 1.36 1.17
query29 12.55 4.05 3.31
query30 0.33 0.12 0.10
query31 2.85 0.64 0.41
query32 3.24 0.60 0.50
query33 3.09 2.97 3.05
query34 16.78 5.18 4.63
query35 4.59 4.69 4.61
query36 0.61 0.51 0.48
query37 0.26 0.09 0.08
query38 0.19 0.06 0.06
query39 0.07 0.05 0.04
query40 0.19 0.17 0.16
query41 0.13 0.06 0.05
query42 0.07 0.05 0.05
query43 0.06 0.06 0.06
Total cold run time: 98.59 s
Total hot run time: 28.12 s
FE Regression Coverage Report
Increment line coverage 61.05% (580/950) :tada:
Increment coverage report
Complete coverage report
Cloud UT Coverage Report
Increment line coverage :tada:
Increment coverage report Complete coverage report
| Category | Coverage |
|---|---|
| Function Coverage | 79.76% (1730/2169) |
| Line Coverage | 65.82% (30641/46555) |
| Region Coverage | 66.59% (15294/22968) |
| Branch Coverage | 56.94% (8135/14288) |
TPC-H: Total hot run time: 36508 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4c300299d2aa5f8be5dd95857bf3247691289cf7, data reload: false
------ Round 1 ----------------------------------
q1 17619 4257 4146 4146
q2 2003 386 238 238
q3 10411 1383 748 748
q4 10376 965 327 327
q5 9842 2174 1984 1984
q6 239 170 135 135
q7 1007 852 726 726
q8 9363 1484 1212 1212
q9 7242 5391 5353 5353
q10 6904 2410 1980 1980
q11 522 321 298 298
q12 668 728 588 588
q13 17788 3731 3037 3037
q14 294 305 274 274
q15 595 516 522 516
q16 696 679 630 630
q17 685 823 559 559
q18 8033 8230 7680 7680
q19 1187 1029 678 678
q20 416 389 270 270
q21 4499 4338 4146 4146
q22 1184 1060 983 983
Total cold run time: 111573 ms
Total hot run time: 36508 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4378 4292 4449 4292
q2 329 426 320 320
q3 2419 2806 2538 2538
q4 1493 2078 1408 1408
q5 4502 4467 4509 4467
q6 214 162 133 133
q7 1982 1913 1868 1868
q8 2712 2469 2437 2437
q9 7503 7611 7250 7250
q10 2975 3112 2674 2674
q11 571 489 475 475
q12 636 746 553 553
q13 3270 3602 3042 3042
q14 263 285 261 261
q15 541 514 500 500
q16 610 634 612 612
q17 1107 1405 1400 1400
q18 7171 7121 7075 7075
q19 874 844 861 844
q20 1905 1949 1810 1810
q21 4716 4311 4169 4169
q22 1093 1059 961 961
Total cold run time: 51264 ms
Total hot run time: 49089 ms