doris
doris copied to clipboard
Published
20 hours ago •
apache
apache
[fix](clone) Fix clone and alter tablet use same tablet path
Proposed changes
Issue Number: close #xxx
The entire process is as follows:
- Drop the tablet.
- Successfully clone the tablet in full.
- Start the incremental clone.
- Start to move the tablet to the trash (the process of actually cleaning the data begins from step 1, where the tablet was dropped).
- The incremental clone fails.
- The incremental clone is successfully retried.
Step 4 moved the data that was just pulled from the full clone to the trash, leading to data loss. The failure in step 5 of the incremental clone was also due to the deletion of the just-pulled snapshot data.
Fix： When cloning, check the tablet status and determine if the tablet directory has already been moved to the trash directory. If it has not been moved to the trash, the clone thread should help move it to the trash directory.
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
TeamCity be ut coverage result: Function Coverage: 35.66% (9013/25278) Line Coverage: 27.29% (74514/273036) Region Coverage: 26.52% (38563/145398) Branch Coverage: 23.37% (19666/84144) Coverage Report: http://coverage.selectdb-in.cc/coverage/8de7e698efd806815a62eff67e7c0a4b9c1d8846_8de7e698efd806815a62eff67e7c0a4b9c1d8846/report/index.html
TPC-DS: Total hot run time: 172537 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8de7e698efd806815a62eff67e7c0a4b9c1d8846, data reload: false
query1 943 376 373 373
query2 6453 2351 2384 2351
query3 6649 201 205 201
query4 19094 17411 17274 17274
query5 4120 418 413 413
query6 254 157 147 147
query7 4580 295 279 279
query8 240 187 185 185
query9 8717 2392 2350 2350
query10 443 273 252 252
query11 10656 10282 10264 10264
query12 126 88 87 87
query13 1633 359 353 353
query14 9257 7510 7498 7498
query15 233 169 175 169
query16 8091 260 251 251
query17 1861 534 526 526
query18 2075 302 272 272
query19 213 151 168 151
query20 93 86 84 84
query21 197 139 138 138
query22 4219 3967 4140 3967
query23 33525 32906 32920 32906
query24 11145 2953 2832 2832
query25 580 349 355 349
query26 1225 151 154 151
query27 3035 318 326 318
query28 7710 2032 2057 2032
query29 860 595 597 595
query30 302 174 179 174
query31 957 756 760 756
query32 95 77 51 51
query33 774 263 250 250
query34 1042 481 474 474
query35 742 605 587 587
query36 1052 871 904 871
query37 135 68 70 68
query38 2917 2810 2791 2791
query39 841 785 777 777
query40 196 122 121 121
query41 46 46 44 44
query42 103 98 95 95
query43 585 551 569 551
query44 1167 722 750 722
query45 177 167 164 164
query46 1066 719 697 697
query47 1853 1736 1764 1736
query48 367 301 288 288
query49 1125 391 383 383
query50 767 381 376 376
query51 6824 6901 6730 6730
query52 98 93 88 88
query53 348 279 285 279
query54 881 424 429 424
query55 74 73 74 73
query56 253 240 246 240
query57 1144 1053 1058 1053
query58 257 208 212 208
query59 3471 3082 3131 3082
query60 265 247 246 246
query61 107 103 108 103
query62 665 470 435 435
query63 310 285 282 282
query64 8996 2203 1693 1693
query65 3218 3083 3107 3083
query66 1401 327 322 322
query67 15359 14865 14726 14726
query68 4584 523 524 523
query69 485 275 265 265
query70 1127 1107 1101 1101
query71 394 267 263 263
query72 7438 5641 5289 5289
query73 703 320 313 313
query74 6091 5714 5704 5704
query75 3302 2656 2587 2587
query76 2554 948 1002 948
query77 416 262 266 262
query78 10312 9839 9905 9839
query79 2284 512 508 508
query80 978 437 425 425
query81 531 246 238 238
query82 632 94 92 92
query83 246 176 171 171
query84 236 86 87 86
query85 1749 310 303 303
query86 487 307 306 306
query87 3308 3201 3104 3104
query88 4181 2339 2342 2339
query89 479 389 395 389
query90 1987 187 186 186
query91 129 107 103 103
query92 66 50 52 50
query93 2107 512 490 490
query94 1205 190 188 188
query95 406 306 310 306
query96 591 265 268 265
query97 3164 3008 3081 3008
query98 324 219 217 217
query99 1074 858 862 858
Total cold run time: 271777 ms
Total hot run time: 172537 ms
ClickBench: Total hot run time: 30.25 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8de7e698efd806815a62eff67e7c0a4b9c1d8846, data reload: false
query1 0.04 0.04 0.02
query2 0.08 0.04 0.04
query3 0.23 0.05 0.04
query4 1.66 0.08 0.10
query5 0.51 0.50 0.53
query6 1.11 0.73 0.72
query7 0.02 0.02 0.01
query8 0.05 0.03 0.04
query9 0.52 0.48 0.48
query10 0.55 0.54 0.52
query11 0.16 0.11 0.12
query12 0.14 0.12 0.12
query13 0.59 0.58 0.59
query14 0.78 0.79 0.76
query15 0.83 0.80 0.81
query16 0.37 0.36 0.38
query17 0.95 1.00 0.95
query18 0.21 0.26 0.24
query19 1.75 1.77 1.71
query20 0.01 0.01 0.01
query21 15.70 0.66 0.65
query22 3.99 7.70 1.72
query23 18.26 1.46 1.31
query24 1.62 0.25 0.24
query25 0.16 0.08 0.08
query26 0.26 0.16 0.17
query27 0.07 0.08 0.08
query28 13.40 1.01 0.99
query29 12.96 3.29 3.29
query30 0.24 0.06 0.06
query31 2.85 0.39 0.38
query32 3.29 0.47 0.47
query33 2.91 2.90 2.86
query34 17.10 4.35 4.38
query35 4.49 4.50 4.53
query36 0.66 0.45 0.45
query37 0.17 0.16 0.15
query38 0.15 0.16 0.15
query39 0.05 0.03 0.03
query40 0.17 0.15 0.14
query41 0.09 0.05 0.04
query42 0.05 0.04 0.05
query43 0.05 0.03 0.04
Total cold run time: 109.25 s
Total hot run time: 30.25 s
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
TPC-H: Total hot run time: 42083 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 15b7664728e01ca4816764a1008c25ff9b406a9c, data reload: false
------ Round 1 ----------------------------------
q1 17603 4312 4299 4299
q2 2001 192 208 192
q3 10437 1207 1225 1207
q4 10185 774 798 774
q5 7472 2695 2746 2695
q6 229 137 136 136
q7 956 617 604 604
q8 9234 2119 2083 2083
q9 9462 6673 6734 6673
q10 9129 3968 3887 3887
q11 456 248 253 248
q12 576 235 240 235
q13 17216 3252 3234 3234
q14 263 228 245 228
q15 523 471 478 471
q16 511 401 416 401
q17 979 716 666 666
q18 8285 7865 7867 7865
q19 6675 1567 1510 1510
q20 638 319 319 319
q21 5199 4065 4132 4065
q22 363 291 308 291
Total cold run time: 118392 ms
Total hot run time: 42083 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4508 4422 4439 4422
q2 376 286 269 269
q3 3191 2876 2900 2876
q4 2048 1746 1616 1616
q5 5301 5505 5517 5505
q6 222 122 125 122
q7 2259 1851 1850 1850
q8 3235 3387 3409 3387
q9 8664 8630 8723 8630
q10 4127 3764 3715 3715
q11 582 517 504 504
q12 802 634 634 634
q13 16020 3158 3155 3155
q14 298 272 284 272
q15 526 498 473 473
q16 515 440 440 440
q17 1823 1507 1525 1507
q18 8072 7501 7464 7464
q19 1650 1556 1583 1556
q20 2002 1843 1834 1834
q21 8735 4769 4841 4769
q22 575 525 505 505
Total cold run time: 75531 ms
Total hot run time: 55505 ms
TeamCity be ut coverage result: Function Coverage: 35.66% (9015/25278) Line Coverage: 27.30% (74540/273047) Region Coverage: 26.53% (38581/145418) Branch Coverage: 23.39% (19684/84160) Coverage Report: http://coverage.selectdb-in.cc/coverage/15b7664728e01ca4816764a1008c25ff9b406a9c_15b7664728e01ca4816764a1008c25ff9b406a9c/report/index.html
TPC-DS: Total hot run time: 168084 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 15b7664728e01ca4816764a1008c25ff9b406a9c, data reload: false
query1 911 379 377 377
query2 6435 2486 2204 2204
query3 6644 208 209 208
query4 19990 17254 17289 17254
query5 4091 431 418 418
query6 245 152 161 152
query7 4591 296 296 296
query8 235 187 183 183
query9 8602 2362 2359 2359
query10 461 336 254 254
query11 10624 10158 9997 9997
query12 133 91 88 88
query13 1662 374 353 353
query14 10091 6919 6227 6227
query15 216 175 167 167
query16 7593 275 265 265
query17 1297 524 527 524
query18 1787 275 266 266
query19 200 175 159 159
query20 95 87 85 85
query21 190 131 128 128
query22 4157 3994 3827 3827
query23 33750 32846 33001 32846
query24 7014 2938 2852 2852
query25 541 362 366 362
query26 705 157 156 156
query27 1940 331 330 330
query28 3791 2060 2065 2060
query29 839 612 598 598
query30 242 170 172 170
query31 958 797 761 761
query32 79 59 59 59
query33 521 292 278 278
query34 867 478 478 478
query35 708 640 581 581
query36 1054 929 925 925
query37 105 68 71 68
query38 2868 2805 2788 2788
query39 814 791 775 775
query40 193 126 123 123
query41 47 43 44 43
query42 104 94 100 94
query43 566 545 542 542
query44 1058 721 741 721
query45 170 160 171 160
query46 1060 696 721 696
query47 1858 1761 1758 1758
query48 396 293 306 293
query49 767 376 378 376
query50 795 400 394 394
query51 6870 6624 6656 6624
query52 102 95 95 95
query53 357 295 293 293
query54 541 469 426 426
query55 77 71 72 71
query56 263 240 238 238
query57 1115 1059 1058 1058
query58 262 212 212 212
query59 3380 3092 3077 3077
query60 270 247 251 247
query61 93 91 109 91
query62 564 447 442 442
query63 310 282 285 282
query64 3350 1757 1731 1731
query65 3169 3094 3090 3090
query66 803 337 322 322
query67 15292 14872 14855 14855
query68 4563 550 547 547
query69 442 272 269 269
query70 1110 1101 1045 1045
query71 362 278 277 277
query72 7444 2735 2616 2616
query73 714 330 324 324
query74 6044 5681 5562 5562
query75 3252 2627 2638 2627
query76 2149 988 996 988
query77 391 267 261 261
query78 10161 9666 9721 9666
query79 2548 513 511 511
query80 1072 440 435 435
query81 523 253 238 238
query82 943 100 94 94
query83 253 173 176 173
query84 255 88 89 88
query85 1043 271 291 271
query86 477 325 299 299
query87 3338 3148 3092 3092
query88 3757 2380 2352 2352
query89 481 396 384 384
query90 2007 191 193 191
query91 133 112 106 106
query92 61 48 55 48
query93 1934 530 520 520
query94 1168 201 197 197
query95 416 324 326 324
query96 582 269 264 264
query97 3143 3035 3019 3019
query98 240 225 221 221
query99 1183 843 845 843
Total cold run time: 253067 ms
Total hot run time: 168084 ms
ClickBench: Total hot run time: 30.57 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 15b7664728e01ca4816764a1008c25ff9b406a9c, data reload: false
query1 0.04 0.03 0.04
query2 0.08 0.03 0.04
query3 0.23 0.05 0.05
query4 1.71 0.07 0.07
query5 0.50 0.49 0.56
query6 1.13 0.72 0.72
query7 0.02 0.01 0.02
query8 0.05 0.04 0.05
query9 0.53 0.52 0.49
query10 0.56 0.55 0.54
query11 0.14 0.11 0.12
query12 0.14 0.12 0.12
query13 0.60 0.58 0.59
query14 0.80 0.78 0.78
query15 0.84 0.81 0.82
query16 0.37 0.34 0.36
query17 0.93 0.95 0.99
query18 0.24 0.23 0.25
query19 1.76 1.69 1.68
query20 0.01 0.02 0.01
query21 15.49 0.69 0.67
query22 4.11 7.59 1.97
query23 18.29 1.32 1.26
query24 2.05 0.22 0.20
query25 0.14 0.08 0.09
query26 0.27 0.16 0.17
query27 0.08 0.07 0.08
query28 13.28 1.02 1.00
query29 13.23 3.30 3.28
query30 0.26 0.06 0.05
query31 2.88 0.39 0.39
query32 3.27 0.47 0.47
query33 2.85 2.87 2.92
query34 17.06 4.47 4.43
query35 4.53 4.52 4.69
query36 0.65 0.50 0.46
query37 0.18 0.15 0.15
query38 0.16 0.16 0.16
query39 0.04 0.04 0.03
query40 0.17 0.14 0.14
query41 0.09 0.05 0.05
query42 0.06 0.05 0.05
query43 0.04 0.03 0.03
Total cold run time: 109.86 s
Total hot run time: 30.57 s