doris
doris copied to clipboard
Published
20 hours ago •
apache
apache
[Fix](group commit) Fix group commit flink error message
When using stream processing frameworks like Flink with group commit mode enabled, the uncertain size of imported data makes such behavior prohibitive. Previously, to simplify the process, the error message for excessive data volume during streamload was combined with the one for group commit mode, leading to confusion for users when encountering errors indicating the data volume is too large during Flink imports. To address this issue, we are adjusting the logic: if a user employs stream processing imports like Flink with group commit mode enabled, we will automatically disable group commit mode, switching to the standard import mode instead. This is the essence of this PR.
Proposed changes
Issue Number: close #xxx
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
PR approved by at least one committer and no changes requested.
TPC-H: Total hot run time: 40842 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 230cf3c298c3849c3337ff9bfde1743d04545208, data reload: false
------ Round 1 ----------------------------------
q1 17698 5000 4895 4895
q2 2042 144 135 135
q3 10576 979 984 979
q4 4657 952 986 952
q5 7617 3125 3166 3125
q6 195 133 136 133
q7 1241 781 759 759
q8 9230 2038 2042 2038
q9 7421 6559 6553 6553
q10 8305 2621 2626 2621
q11 419 203 207 203
q12 805 322 331 322
q13 17953 3629 3588 3588
q14 288 261 263 261
q15 635 510 510 510
q16 480 412 406 406
q17 921 884 867 867
q18 7546 6526 6513 6513
q19 1530 1487 1483 1483
q20 545 281 268 268
q21 6658 3905 3969 3905
q22 868 326 340 326
Total cold run time: 107630 ms
Total hot run time: 40842 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4874 4919 5025 4919
q2 298 189 189 189
q3 3664 3639 3660 3639
q4 2571 2554 2627 2554
q5 5774 5743 5742 5742
q6 210 125 126 125
q7 2258 1637 1631 1631
q8 3034 3107 3115 3107
q9 8754 8719 8673 8673
q10 6789 4232 4224 4224
q11 505 364 354 354
q12 772 537 552 537
q13 4403 3399 3378 3378
q14 277 237 238 237
q15 598 525 504 504
q16 488 465 411 411
q17 1677 1621 1611 1611
q18 8230 7759 7677 7677
q19 1622 1632 1631 1631
q20 2105 1840 1809 1809
q21 6590 6151 6139 6139
q22 606 516 512 512
Total cold run time: 66099 ms
Total hot run time: 59603 ms
TPC-DS: Total hot run time: 176591 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 230cf3c298c3849c3337ff9bfde1743d04545208, data reload: false
query1 926 348 342 342
query2 6513 1732 1709 1709
query3 6693 208 200 200
query4 22969 21202 21066 21066
query5 4306 379 368 368
query6 266 167 168 167
query7 4609 309 284 284
query8 250 191 205 191
query9 8434 2769 2739 2739
query10 414 216 220 216
query11 14891 14597 14564 14564
query12 135 79 82 79
query13 1678 422 425 422
query14 9143 7653 7642 7642
query15 206 181 189 181
query16 7465 256 243 243
query17 1430 545 537 537
query18 1952 269 264 264
query19 195 144 149 144
query20 90 82 81 81
query21 189 121 119 119
query22 4928 4749 4683 4683
query23 32451 31564 31606 31564
query24 12712 3382 3389 3382
query25 637 368 350 350
query26 1877 157 164 157
query27 3041 306 312 306
query28 6634 1813 1808 1808
query29 1142 610 598 598
query30 284 135 145 135
query31 921 744 768 744
query32 93 61 58 58
query33 715 239 229 229
query34 1075 487 494 487
query35 926 828 802 802
query36 997 938 893 893
query37 175 61 62 61
query38 3274 3160 3195 3160
query39 1375 1349 1309 1309
query40 286 106 105 105
query41 41 36 35 35
query42 106 97 101 97
query43 473 449 437 437
query44 1081 684 688 684
query45 200 175 176 175
query46 1041 770 754 754
query47 1589 1576 1574 1574
query48 415 356 351 351
query49 1213 299 311 299
query50 771 364 372 364
query51 4469 4293 4360 4293
query52 107 94 92 92
query53 398 301 292 292
query54 298 215 232 215
query55 85 78 83 78
query56 226 208 195 195
query57 1012 984 966 966
query58 216 195 195 195
query59 2295 2197 2107 2107
query60 237 208 218 208
query61 84 85 89 85
query62 597 400 386 386
query63 323 281 291 281
query64 6381 3101 3092 3092
query65 3308 3277 3228 3228
query66 1363 326 332 326
query67 14484 14357 14304 14304
query68 5075 556 550 550
query69 524 372 351 351
query70 1201 1232 1312 1232
query71 453 257 252 252
query72 6357 2773 2643 2643
query73 688 314 307 307
query74 6898 6526 6415 6415
query75 3181 2563 2575 2563
query76 3242 1121 1206 1121
query77 349 235 234 234
query78 9498 8875 8759 8759
query79 972 514 513 513
query80 517 361 337 337
query81 433 203 199 199
query82 166 84 85 84
query83 138 123 119 119
query84 227 78 78 78
query85 1060 362 347 347
query86 291 299 304 299
query87 3444 3334 3267 3267
query88 2662 2282 2278 2278
query89 428 358 354 354
query90 1959 164 166 164
query91 157 128 132 128
query92 54 50 48 48
query93 1023 523 525 523
query94 1088 179 176 176
query95 423 331 345 331
query96 567 259 268 259
query97 4453 4270 4268 4268
query98 232 208 195 195
query99 1073 695 730 695
Total cold run time: 267943 ms
Total hot run time: 176591 ms
TPC-H: Total hot run time: 41172 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit da62663e6490567568f9f6dd2120bad5db139b32, data reload: false
------ Round 1 ----------------------------------
q1 17654 5015 4937 4937
q2 2045 143 133 133
q3 10572 994 1051 994
q4 4659 948 990 948
q5 7622 3142 3193 3142
q6 196 133 134 133
q7 1260 788 777 777
q8 9257 2075 2056 2056
q9 7503 6580 6572 6572
q10 8316 2666 2650 2650
q11 423 205 205 205
q12 751 334 330 330
q13 17927 3614 3608 3608
q14 293 262 267 262
q15 577 544 505 505
q16 478 405 433 405
q17 932 882 892 882
q18 7387 6611 6601 6601
q19 1545 1483 1482 1482
q20 551 283 261 261
q21 6538 3950 3976 3950
q22 878 346 339 339
Total cold run time: 107364 ms
Total hot run time: 41172 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4906 4874 4866 4866
q2 304 198 196 196
q3 3608 3568 3576 3568
q4 2541 2533 2551 2533
q5 5729 5750 5753 5750
q6 213 128 130 128
q7 2254 1650 1686 1650
q8 3067 3138 3174 3138
q9 8754 8752 8729 8729
q10 6923 4248 4251 4248
q11 524 397 405 397
q12 784 552 559 552
q13 4202 3408 3419 3408
q14 266 238 235 235
q15 592 515 504 504
q16 532 451 497 451
q17 1697 1620 1641 1620
q18 8301 7717 7571 7571
q19 1635 1637 1629 1629
q20 2124 1849 1863 1849
q21 6690 6192 6224 6192
q22 578 517 499 499
Total cold run time: 66224 ms
Total hot run time: 59713 ms
TeamCity be ut coverage result: Function Coverage: 35.71% (8547/23937) Line Coverage: 27.52% (69357/251983) Region Coverage: 26.68% (35968/134837) Branch Coverage: 23.48% (18386/78294) Coverage Report: http://coverage.selectdb-in.cc/coverage/da62663e6490567568f9f6dd2120bad5db139b32_da62663e6490567568f9f6dd2120bad5db139b32/report/index.html
TPC-DS: Total hot run time: 176673 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit da62663e6490567568f9f6dd2120bad5db139b32, data reload: false
query1 912 354 356 354
query2 6519 1830 1703 1703
query3 6702 206 204 204
query4 22976 21156 21107 21107
query5 4229 376 379 376
query6 271 166 167 166
query7 4593 297 293 293
query8 256 188 205 188
query9 8432 2787 2779 2779
query10 405 222 228 222
query11 15113 14602 14389 14389
query12 140 80 87 80
query13 1684 424 414 414
query14 9190 7490 7569 7490
query15 202 186 190 186
query16 7494 259 242 242
query17 1413 572 540 540
query18 1946 289 268 268
query19 189 148 144 144
query20 85 84 81 81
query21 190 118 122 118
query22 5069 4826 4850 4826
query23 32481 31528 31525 31525
query24 12718 3399 3366 3366
query25 641 367 356 356
query26 1867 151 157 151
query27 3049 319 318 318
query28 6580 1815 1811 1811
query29 1115 617 619 617
query30 279 142 147 142
query31 922 757 773 757
query32 102 56 54 54
query33 721 228 230 228
query34 1075 475 489 475
query35 926 826 826 826
query36 1011 923 880 880
query37 167 60 63 60
query38 3330 3185 3230 3185
query39 1380 1337 1309 1309
query40 284 108 106 106
query41 38 37 35 35
query42 107 97 99 97
query43 485 442 450 442
query44 1056 675 689 675
query45 197 181 174 174
query46 1030 789 762 762
query47 1673 1553 1582 1553
query48 424 368 340 340
query49 1201 304 301 301
query50 774 375 381 375
query51 4413 4350 4323 4323
query52 105 96 95 95
query53 401 305 306 305
query54 302 220 227 220
query55 86 82 80 80
query56 237 196 204 196
query57 1097 956 957 956
query58 221 194 195 194
query59 2376 2268 2136 2136
query60 247 215 235 215
query61 85 84 86 84
query62 601 404 370 370
query63 317 288 283 283
query64 6371 3068 3115 3068
query65 3311 3259 3247 3247
query66 1354 327 321 321
query67 14663 14463 14256 14256
query68 5052 549 541 541
query69 517 353 363 353
query70 1265 1150 1216 1150
query71 440 249 253 249
query72 6586 2821 2635 2635
query73 697 310 311 310
query74 6907 6513 6502 6502
query75 3233 2592 2552 2552
query76 3224 1129 1211 1129
query77 356 238 234 234
query78 9535 8867 8893 8867
query79 976 510 508 508
query80 523 352 344 344
query81 437 202 207 202
query82 167 88 82 82
query83 137 124 117 117
query84 226 79 79 79
query85 1031 361 349 349
query86 294 311 295 295
query87 3506 3334 3328 3328
query88 2756 2291 2284 2284
query89 429 368 358 358
query90 1953 164 164 164
query91 148 131 131 131
query92 55 46 53 46
query93 1022 516 488 488
query94 1169 182 176 176
query95 424 332 330 330
query96 573 262 264 262
query97 4490 4270 4293 4270
query98 216 211 196 196
query99 1041 721 703 703
Total cold run time: 269215 ms
Total hot run time: 176673 ms
ClickBench: Total hot run time: 30.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit da62663e6490567568f9f6dd2120bad5db139b32, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.02
query3 0.22 0.07 0.07
query4 1.64 0.10 0.10
query5 0.48 0.49 0.49
query6 1.34 0.60 0.61
query7 0.01 0.01 0.02
query8 0.04 0.02 0.03
query9 0.52 0.46 0.45
query10 0.49 0.52 0.48
query11 0.13 0.10 0.10
query12 0.12 0.09 0.10
query13 0.58 0.59 0.58
query14 0.78 0.77 0.78
query15 0.82 0.81 0.79
query16 0.33 0.33 0.33
query17 0.90 0.93 0.91
query18 0.17 0.15 0.16
query19 1.76 1.72 1.66
query20 0.01 0.02 0.02
query21 15.42 0.62 0.61
query22 2.83 4.14 1.83
query23 17.60 1.06 0.96
query24 2.01 0.75 0.25
query25 0.60 0.08 0.05
query26 0.16 0.14 0.13
query27 0.05 0.05 0.05
query28 11.97 0.84 0.83
query29 12.57 3.32 3.26
query30 0.52 0.48 0.50
query31 2.77 0.37 0.37
query32 3.33 0.47 0.48
query33 3.13 3.18 3.14
query34 15.35 4.55 4.54
query35 4.55 4.54 4.48
query36 1.07 0.95 0.94
query37 0.08 0.05 0.05
query38 0.04 0.03 0.04
query39 0.02 0.02 0.02
query40 0.16 0.15 0.15
query41 0.07 0.02 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 104.77 s
Total hot run time: 30.35 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit da62663e6490567568f9f6dd2120bad5db139b32 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 16.6 seconds inserted 10000000 Rows, about 602K ops/s
PR approved by at least one committer and no changes requested.