doris
doris copied to clipboard
apache
[fix](compaction) generate single compaction and cumu(base) compaction tasks simultaneously.
BUG
Each time a compaction task was generated, either a single compaction or a cumu (base) compaction task. Because there is no matched version available to pull from the peer, it keeps generating single compaction tasks, cumu (base) compaction tasks cannot be generated.
HOW TO FIX
- When single compaction fails to complete, skip the compaction task for this tablet temporarily.
- Each time a compaction task is generated, create two tasks: one for single compaction and one for cumu(base) compaction.
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
TPC-H: Total hot run time: 41337 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 909f62973fe7940b3c3603e6bec09f33a49f24d7, data reload: false
------ Round 1 ----------------------------------
q1 17885 4319 4251 4251
q2 2014 193 187 187
q3 10471 1251 1204 1204
q4 10193 815 784 784
q5 7530 2858 2683 2683
q6 221 130 132 130
q7 972 570 566 566
q8 9238 2171 2118 2118
q9 9459 6840 6755 6755
q10 9606 3932 3886 3886
q11 422 279 244 244
q12 515 222 219 219
q13 18207 3177 3210 3177
q14 274 222 229 222
q15 522 473 477 473
q16 527 420 407 407
q17 973 707 772 707
q18 8351 7725 7771 7725
q19 3351 1568 1558 1558
q20 664 330 326 326
q21 5359 3433 4271 3433
q22 348 283 282 282
Total cold run time: 117102 ms
Total hot run time: 41337 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4562 4389 4427 4389
q2 381 269 276 269
q3 3198 2965 2954 2954
q4 1894 1658 1566 1566
q5 5543 5549 5554 5549
q6 219 124 132 124
q7 1821 1488 1483 1483
q8 3315 3407 3446 3407
q9 8815 8829 8916 8829
q10 4079 3729 3773 3729
q11 588 503 496 496
q12 816 627 636 627
q13 17246 3182 3175 3175
q14 331 290 291 290
q15 537 487 505 487
q16 486 448 458 448
q17 1794 1515 1496 1496
q18 7851 7542 7466 7466
q19 1632 1526 1486 1486
q20 1976 1766 1768 1766
q21 8218 4862 4886 4862
q22 567 489 467 467
Total cold run time: 75869 ms
Total hot run time: 55365 ms
TeamCity be ut coverage result: Function Coverage: 35.59% (8934/25100) Line Coverage: 27.22% (73670/270662) Region Coverage: 26.41% (38060/144114) Branch Coverage: 23.19% (19397/83654) Coverage Report: http://coverage.selectdb-in.cc/coverage/909f62973fe7940b3c3603e6bec09f33a49f24d7_909f62973fe7940b3c3603e6bec09f33a49f24d7/report/index.html
TPC-DS: Total hot run time: 186942 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 909f62973fe7940b3c3603e6bec09f33a49f24d7, data reload: false
query1 909 363 353 353
query2 7575 2462 2459 2459
query3 6697 220 220 220
query4 25396 21226 21297 21226
query5 4112 414 417 414
query6 264 184 201 184
query7 4581 279 278 278
query8 243 193 183 183
query9 8581 2318 2305 2305
query10 437 249 259 249
query11 14652 14099 14299 14099
query12 145 90 86 86
query13 1641 356 371 356
query14 9117 7888 7926 7888
query15 260 175 188 175
query16 7997 269 262 262
query17 1747 563 582 563
query18 2094 282 277 277
query19 203 155 152 152
query20 90 89 87 87
query21 194 130 128 128
query22 4970 4819 4789 4789
query23 33940 33227 33008 33008
query24 10857 2990 3058 2990
query25 629 389 393 389
query26 1186 160 150 150
query27 2596 323 323 323
query28 7025 2019 1985 1985
query29 879 620 609 609
query30 241 155 149 149
query31 970 756 759 756
query32 92 59 57 57
query33 746 257 253 253
query34 1012 489 472 472
query35 819 729 684 684
query36 1100 914 942 914
query37 134 68 72 68
query38 3201 2982 3036 2982
query39 1597 1514 1551 1514
query40 204 134 129 129
query41 44 41 42 41
query42 106 103 100 100
query43 565 562 526 526
query44 1223 723 739 723
query45 290 277 270 270
query46 1073 732 719 719
query47 1940 1826 1877 1826
query48 355 297 294 294
query49 868 473 399 399
query50 796 404 374 374
query51 6810 6946 6739 6739
query52 106 89 96 89
query53 357 280 281 280
query54 298 240 241 240
query55 81 74 71 71
query56 236 218 215 215
query57 1176 1132 1107 1107
query58 214 196 199 196
query59 3479 3270 3404 3270
query60 257 229 234 229
query61 94 104 89 89
query62 626 447 466 447
query63 304 279 289 279
query64 8494 7171 7234 7171
query65 3070 3033 3015 3015
query66 768 357 332 332
query67 15670 15226 15306 15226
query68 9902 544 545 544
query69 600 315 331 315
query70 1323 1069 1118 1069
query71 516 264 274 264
query72 8663 2642 2433 2433
query73 1565 325 326 325
query74 6490 6191 6044 6044
query75 4832 2687 2678 2678
query76 6001 1028 954 954
query77 712 270 267 267
query78 11041 10235 10219 10219
query79 11144 531 539 531
query80 1544 434 463 434
query81 491 221 227 221
query82 182 98 98 98
query83 202 166 173 166
query84 267 84 82 82
query85 1021 267 256 256
query86 344 298 273 273
query87 3330 3142 3085 3085
query88 4625 2306 2302 2302
query89 527 389 374 374
query90 2037 187 187 187
query91 127 103 132 103
query92 59 48 51 48
query93 5479 519 503 503
query94 1341 189 185 185
query95 418 316 317 316
query96 610 268 275 268
query97 3176 2925 2964 2925
query98 244 225 223 223
query99 1054 818 847 818
Total cold run time: 308300 ms
Total hot run time: 186942 ms
TPC-H: Total hot run time: 41288 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f892b3a6622b8d2c1701a09c54b8951f1692f5a, data reload: false
------ Round 1 ----------------------------------
q1 17606 4307 4254 4254
q2 2019 193 194 193
q3 10567 1217 1132 1132
q4 10909 767 792 767
q5 7763 2685 2687 2685
q6 225 141 137 137
q7 1032 609 592 592
q8 9370 2155 2103 2103
q9 10164 7340 7192 7192
q10 9056 3741 3733 3733
q11 444 244 232 232
q12 503 216 230 216
q13 18607 2944 2966 2944
q14 255 222 226 222
q15 513 474 473 473
q16 515 411 376 376
q17 980 707 745 707
q18 8028 7445 7348 7348
q19 1645 1547 1514 1514
q20 649 321 314 314
q21 5081 3878 4073 3878
q22 355 276 286 276
Total cold run time: 116286 ms
Total hot run time: 41288 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4375 4239 4227 4227
q2 364 288 280 280
q3 3013 2748 2730 2730
q4 1844 1613 1594 1594
q5 5293 5261 5280 5261
q6 213 122 125 122
q7 2292 1919 1887 1887
q8 3205 3358 3358 3358
q9 9139 9104 9054 9054
q10 3926 3718 3669 3669
q11 599 481 480 480
q12 757 604 598 598
q13 17319 2953 2949 2949
q14 298 287 272 272
q15 505 484 479 479
q16 466 412 408 408
q17 1761 1479 1475 1475
q18 7578 7519 7421 7421
q19 2265 1649 1603 1603
q20 1944 1756 1772 1756
q21 4930 4909 4795 4795
q22 570 509 507 507
Total cold run time: 72656 ms
Total hot run time: 54925 ms
TeamCity be ut coverage result: Function Coverage: 35.66% (8978/25180) Line Coverage: 27.33% (74135/271252) Region Coverage: 26.55% (38298/144227) Branch Coverage: 23.35% (19507/83550) Coverage Report: http://coverage.selectdb-in.cc/coverage/2f892b3a6622b8d2c1701a09c54b8951f1692f5a_2f892b3a6622b8d2c1701a09c54b8951f1692f5a/report/index.html
TPC-DS: Total hot run time: 186934 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f892b3a6622b8d2c1701a09c54b8951f1692f5a, data reload: false
query1 925 357 347 347
query2 6472 2438 2399 2399
query3 6649 203 207 203
query4 23112 21085 21294 21085
query5 4191 433 442 433
query6 270 185 177 177
query7 4611 297 289 289
query8 250 212 186 186
query9 8391 2356 2356 2356
query10 435 262 257 257
query11 14886 14094 14195 14094
query12 143 96 92 92
query13 1647 386 374 374
query14 9636 7694 7599 7599
query15 275 175 183 175
query16 8194 266 266 266
query17 1845 590 572 572
query18 2123 290 280 280
query19 213 165 159 159
query20 97 90 89 89
query21 200 131 134 131
query22 5089 4843 4761 4761
query23 33895 33414 33384 33384
query24 12126 3017 2921 2921
query25 667 391 387 387
query26 1767 164 155 155
query27 3015 371 322 322
query28 7577 2039 2034 2034
query29 1012 599 606 599
query30 286 154 154 154
query31 961 747 743 743
query32 96 52 56 52
query33 757 258 250 250
query34 1055 482 491 482
query35 803 670 680 670
query36 1081 945 923 923
query37 291 69 68 68
query38 3225 2976 3009 2976
query39 1613 1527 1549 1527
query40 274 129 133 129
query41 41 40 41 40
query42 103 94 100 94
query43 602 523 547 523
query44 1238 726 726 726
query45 265 246 251 246
query46 1095 744 733 733
query47 1909 1842 1857 1842
query48 377 302 300 300
query49 1216 394 401 394
query50 786 388 396 388
query51 6840 6736 6653 6653
query52 102 95 90 90
query53 361 284 283 283
query54 310 235 244 235
query55 81 78 76 76
query56 245 226 224 224
query57 1209 1113 1148 1113
query58 237 206 202 202
query59 3559 3406 3342 3342
query60 266 232 237 232
query61 111 90 88 88
query62 652 480 457 457
query63 311 286 289 286
query64 9621 7221 7220 7220
query65 3114 3082 3100 3082
query66 1366 355 352 352
query67 15524 15026 15025 15025
query68 6301 543 525 525
query69 524 316 310 310
query70 1236 1164 1156 1156
query71 480 277 273 273
query72 8271 2608 2405 2405
query73 717 325 326 325
query74 6504 6072 5996 5996
query75 3910 2683 2643 2643
query76 4206 1048 1035 1035
query77 637 270 270 270
query78 10920 10290 10165 10165
query79 8434 516 527 516
query80 1851 451 449 449
query81 524 223 233 223
query82 1483 95 99 95
query83 334 167 180 167
query84 269 85 86 85
query85 1545 307 268 268
query86 473 312 309 309
query87 3244 3090 3077 3077
query88 5121 2322 2334 2322
query89 563 392 391 391
query90 2012 188 190 188
query91 130 101 98 98
query92 62 50 49 49
query93 6768 518 503 503
query94 1230 181 186 181
query95 383 303 306 303
query96 595 267 263 263
query97 3093 2931 2967 2931
query98 241 228 217 217
query99 1299 894 850 850
Total cold run time: 305056 ms
Total hot run time: 186934 ms
TeamCity be ut coverage result: Function Coverage: 35.73% (8993/25171) Line Coverage: 27.40% (74373/271433) Region Coverage: 26.64% (38453/144365) Branch Coverage: 23.46% (19622/83636) Coverage Report: http://coverage.selectdb-in.cc/coverage/a1d3814483f3a3bd9b2c1580b7c203afa4bf2fba_a1d3814483f3a3bd9b2c1580b7c203afa4bf2fba/report/index.html
PR approved by at least one committer and no changes requested.
TeamCity be ut coverage result: Function Coverage: 35.72% (8993/25173) Line Coverage: 27.40% (74415/271592) Region Coverage: 26.62% (38461/144459) Branch Coverage: 23.45% (19624/83672) Coverage Report: http://coverage.selectdb-in.cc/coverage/f9d800a8c2e00b54b53f3a78ff31f2df2fd59202_f9d800a8c2e00b54b53f3a78ff31f2df2fd59202/report/index.html
TeamCity be ut coverage result: Function Coverage: 35.70% (8992/25191) Line Coverage: 27.38% (74411/271764) Region Coverage: 26.61% (38460/144532) Branch Coverage: 23.44% (19624/83730) Coverage Report: http://coverage.selectdb-in.cc/coverage/caea0be98fe466b7e9cafde1d57899e7e20387b9_caea0be98fe466b7e9cafde1d57899e7e20387b9/report/index.html
TeamCity be ut coverage result: Function Coverage: 35.69% (8996/25203) Line Coverage: 27.37% (74419/271908) Region Coverage: 26.60% (38471/144615) Branch Coverage: 23.42% (19621/83790) Coverage Report: http://coverage.selectdb-in.cc/coverage/b105c1d1cde4701105fa735aca0e23bb95b68b7b_b105c1d1cde4701105fa735aca0e23bb95b68b7b/report/index.html
TPC-H: Total hot run time: 41946 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b105c1d1cde4701105fa735aca0e23bb95b68b7b, data reload: false
------ Round 1 ----------------------------------
q1 17596 4288 4192 4192
q2 2014 197 193 193
q3 10469 1248 1152 1152
q4 10213 725 824 725
q5 7476 2727 2738 2727
q6 224 131 141 131
q7 1015 607 596 596
q8 9233 2146 2106 2106
q9 9128 6769 6680 6680
q10 9347 3967 3943 3943
q11 454 253 255 253
q12 527 235 236 235
q13 17261 3164 3343 3164
q14 257 217 219 217
q15 508 485 472 472
q16 490 411 406 406
q17 1003 722 747 722
q18 8342 7796 7723 7723
q19 3055 1584 1492 1492
q20 650 332 336 332
q21 5338 4184 4190 4184
q22 379 304 301 301
Total cold run time: 114979 ms
Total hot run time: 41946 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4583 4363 4468 4363
q2 379 265 271 265
q3 3095 2951 2757 2757
q4 1893 1656 1659 1656
q5 5534 5467 5478 5467
q6 217 132 129 129
q7 2337 1948 1973 1948
q8 3244 3362 3391 3362
q9 8709 8741 8601 8601
q10 3917 3860 3927 3860
q11 595 491 498 491
q12 804 621 615 615
q13 16169 3172 3207 3172
q14 302 252 248 248
q15 523 468 471 468
q16 467 420 408 408
q17 1761 1482 1442 1442
q18 7665 7758 7472 7472
q19 1659 1591 1542 1542
q20 1973 1766 1755 1755
q21 9359 4860 4962 4860
q22 577 525 502 502
Total cold run time: 75762 ms
Total hot run time: 55383 ms
TPC-DS: Total hot run time: 187071 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b105c1d1cde4701105fa735aca0e23bb95b68b7b, data reload: false
query1 914 372 349 349
query2 6431 2261 2347 2261
query3 6640 212 209 209
query4 23355 21153 21289 21153
query5 4168 412 421 412
query6 278 167 182 167
query7 4593 289 291 289
query8 236 182 203 182
query9 8649 2427 2419 2419
query10 440 251 254 251
query11 14845 14187 14166 14166
query12 133 88 92 88
query13 1645 359 371 359
query14 9650 8466 7711 7711
query15 223 177 169 169
query16 7948 277 271 271
query17 1705 547 545 545
query18 2054 273 274 273
query19 199 148 149 148
query20 89 86 82 82
query21 193 128 126 126
query22 5048 4821 4738 4738
query23 34062 33431 33454 33431
query24 7663 2911 2906 2906
query25 572 387 352 352
query26 699 154 154 154
query27 2178 319 332 319
query28 5477 2075 2072 2072
query29 881 609 591 591
query30 226 159 151 151
query31 997 755 737 737
query32 95 50 54 50
query33 536 251 240 240
query34 874 483 483 483
query35 768 685 662 662
query36 1041 907 937 907
query37 106 71 66 66
query38 2873 2798 2776 2776
query39 1604 1555 1548 1548
query40 197 125 126 125
query41 44 41 41 41
query42 104 94 95 94
query43 563 555 545 545
query44 1068 743 734 734
query45 280 252 260 252
query46 1098 695 701 695
query47 1977 1925 1912 1912
query48 383 304 303 303
query49 866 416 405 405
query50 769 385 393 385
query51 6958 6819 6847 6819
query52 105 89 96 89
query53 358 283 282 282
query54 552 443 450 443
query55 74 73 71 71
query56 248 231 237 231
query57 1238 1173 1149 1149
query58 221 203 204 203
query59 3492 3312 2965 2965
query60 260 243 242 242
query61 107 107 109 107
query62 623 478 480 478
query63 305 287 282 282
query64 8505 7430 7507 7430
query65 3120 3077 3114 3077
query66 799 349 368 349
query67 15370 14968 14919 14919
query68 4599 539 538 538
query69 498 312 315 312
query70 1166 1162 1127 1127
query71 386 279 267 267
query72 7373 2677 2489 2489
query73 719 328 329 328
query74 6529 6104 6109 6104
query75 3308 2647 2611 2611
query76 2322 1089 1024 1024
query77 398 264 257 257
query78 10709 10011 10069 10011
query79 3578 522 511 511
query80 2072 436 434 434
query81 516 215 215 215
query82 774 92 88 88
query83 287 170 162 162
query84 264 115 85 85
query85 1897 271 259 259
query86 468 316 310 310
query87 3276 3157 3080 3080
query88 4778 2402 2430 2402
query89 492 379 373 373
query90 2062 184 186 184
query91 123 96 96 96
query92 57 47 46 46
query93 5347 523 509 509
query94 1302 179 194 179
query95 393 309 302 302
query96 601 276 266 266
query97 3178 2990 2958 2958
query98 232 210 209 209
query99 1275 914 895 895
Total cold run time: 280751 ms
Total hot run time: 187071 ms
PR approved by at least one committer and no changes requested.