doris
doris copied to clipboard
Published
20 hours ago •
apache
apache
[enhancement](export) filter empty partition before export table to remote storage
Proposed changes
Issue Number: close #xxx
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
PR approved by at least one committer and no changes requested.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
TPC-DS: Total hot run time: 183021 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ec3272b375a30e3c309cc60655ccee1fd7c19ef8, data reload: false
query1 916 394 366 366
query2 6268 2413 2350 2350
query3 6644 212 214 212
query4 26321 21867 22018 21867
query5 3798 411 418 411
query6 262 162 167 162
query7 4616 298 296 296
query8 238 203 210 203
query9 8471 2493 2463 2463
query10 434 254 262 254
query11 15285 14740 14892 14740
query12 121 85 86 85
query13 1588 358 369 358
query14 10398 7844 8447 7844
query15 258 169 165 165
query16 8387 263 255 255
query17 1755 553 537 537
query18 2088 262 268 262
query19 245 148 148 148
query20 91 84 84 84
query21 192 124 125 124
query22 5065 4865 4841 4841
query23 33986 33713 33632 33632
query24 10655 2898 2910 2898
query25 598 363 365 363
query26 1315 157 153 153
query27 2887 321 324 321
query28 7415 2033 2027 2027
query29 864 607 611 607
query30 254 167 168 167
query31 969 751 713 713
query32 88 53 53 53
query33 737 241 246 241
query34 1062 487 475 475
query35 803 665 652 652
query36 1064 935 906 906
query37 137 71 70 70
query38 2910 2796 2787 2787
query39 1591 1534 1559 1534
query40 200 128 127 127
query41 49 45 44 44
query42 105 98 99 98
query43 580 539 531 531
query44 1186 743 739 739
query45 274 264 263 263
query46 1080 726 697 697
query47 1919 1880 1892 1880
query48 374 304 301 301
query49 911 401 400 400
query50 764 386 388 386
query51 6920 6778 6810 6778
query52 105 91 91 91
query53 354 289 333 289
query54 891 425 413 413
query55 73 74 74 74
query56 237 227 232 227
query57 1231 1143 1157 1143
query58 221 215 200 200
query59 3546 3162 2982 2982
query60 264 240 226 226
query61 92 89 84 84
query62 641 495 467 467
query63 314 288 285 285
query64 8865 2236 1757 1757
query65 3161 3075 3107 3075
query66 1359 355 333 333
query67 15365 15107 14816 14816
query68 4530 529 515 515
query69 468 298 306 298
query70 1075 1128 1087 1087
query71 410 269 270 269
query72 7132 2573 2379 2379
query73 717 316 321 316
query74 6610 6112 6226 6112
query75 3393 2599 2606 2599
query76 2720 1006 1000 1000
query77 393 269 268 268
query78 10592 10314 10313 10313
query79 2481 512 522 512
query80 1097 428 425 425
query81 544 243 246 243
query82 750 90 99 90
query83 243 169 167 167
query84 234 83 81 81
query85 1620 267 262 262
query86 558 284 302 284
query87 3322 3121 3113 3113
query88 4084 2331 2331 2331
query89 466 367 378 367
query90 1946 196 184 184
query91 128 96 94 94
query92 64 48 107 48
query93 1832 479 479 479
query94 1131 174 173 173
query95 379 309 296 296
query96 581 277 264 264
query97 3203 3022 3014 3014
query98 236 216 208 208
query99 1121 907 898 898
Total cold run time: 285917 ms
Total hot run time: 183021 ms
ClickBench: Total hot run time: 30.37 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ec3272b375a30e3c309cc60655ccee1fd7c19ef8, data reload: false
query1 0.04 0.03 0.04
query2 0.09 0.04 0.04
query3 0.22 0.05 0.05
query4 1.69 0.08 0.07
query5 0.50 0.50 0.50
query6 1.12 0.72 0.73
query7 0.02 0.01 0.02
query8 0.05 0.05 0.04
query9 0.53 0.50 0.49
query10 0.55 0.55 0.55
query11 0.16 0.11 0.10
query12 0.15 0.12 0.12
query13 0.60 0.60 0.59
query14 0.78 0.79 0.76
query15 0.82 0.80 0.81
query16 0.37 0.38 0.37
query17 0.99 0.94 0.94
query18 0.22 0.26 0.26
query19 1.76 1.70 1.69
query20 0.02 0.01 0.00
query21 15.52 0.70 0.66
query22 4.75 7.28 1.69
query23 18.32 1.32 1.26
query24 1.55 0.28 0.26
query25 0.15 0.09 0.07
query26 0.27 0.17 0.17
query27 0.08 0.08 0.07
query28 13.37 1.02 1.00
query29 13.59 3.49 3.40
query30 0.24 0.06 0.05
query31 2.86 0.38 0.39
query32 3.31 0.47 0.46
query33 2.89 2.87 2.89
query34 17.12 4.43 4.36
query35 4.53 4.49 4.63
query36 0.66 0.47 0.45
query37 0.17 0.15 0.15
query38 0.16 0.15 0.15
query39 0.05 0.04 0.04
query40 0.17 0.14 0.15
query41 0.09 0.05 0.04
query42 0.05 0.04 0.05
query43 0.04 0.04 0.04
Total cold run time: 110.62 s
Total hot run time: 30.37 s
TPC-H: Total hot run time: 40094 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d4e21fcc6d1fa35b1768c8402621513e7209e849, data reload: false
------ Round 1 ----------------------------------
q1 18001 4414 4504 4414
q2 2600 205 205 205
q3 11502 1269 1177 1177
q4 10989 844 788 788
q5 7498 2691 2766 2691
q6 225 131 134 131
q7 947 601 603 601
q8 9580 2049 2029 2029
q9 9040 6446 6412 6412
q10 8885 3694 3669 3669
q11 450 239 241 239
q12 447 219 210 210
q13 17764 3004 2967 2967
q14 275 211 223 211
q15 511 469 481 469
q16 532 376 379 376
q17 956 691 718 691
q18 7974 7467 7421 7421
q19 5452 1529 1478 1478
q20 657 313 306 306
q21 4929 3896 3330 3330
q22 394 279 280 279
Total cold run time: 119608 ms
Total hot run time: 40094 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4288 4196 4193 4193
q2 363 263 269 263
q3 2973 2719 2689 2689
q4 1857 1580 1562 1562
q5 5277 5265 5271 5265
q6 210 122 123 122
q7 2127 1737 1751 1737
q8 3192 3288 3286 3286
q9 8322 8334 8374 8334
q10 3883 3711 3747 3711
q11 603 499 486 486
q12 756 612 598 598
q13 16438 2972 2981 2972
q14 305 267 265 265
q15 508 484 478 478
q16 473 415 432 415
q17 1779 1454 1469 1454
q18 7590 7547 7413 7413
q19 1649 1585 1451 1451
q20 1955 1754 1779 1754
q21 4861 4772 4716 4716
q22 579 484 486 484
Total cold run time: 69988 ms
Total hot run time: 53648 ms
TPC-DS: Total hot run time: 168772 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d4e21fcc6d1fa35b1768c8402621513e7209e849, data reload: false
query1 917 374 370 370
query2 6466 2490 2391 2391
query3 6650 207 206 206
query4 18975 17180 17299 17180
query5 4137 424 434 424
query6 245 160 157 157
query7 4592 304 298 298
query8 236 185 186 185
query9 8479 2395 2376 2376
query10 458 279 295 279
query11 11684 10116 9960 9960
query12 137 93 85 85
query13 1642 348 357 348
query14 10132 7051 6744 6744
query15 219 170 169 169
query16 7809 256 261 256
query17 1817 519 519 519
query18 1897 274 267 267
query19 199 157 156 156
query20 95 89 102 89
query21 200 130 138 130
query22 4341 3836 3991 3836
query23 33608 32918 33180 32918
query24 11989 2751 2821 2751
query25 661 355 360 355
query26 1799 151 150 150
query27 3032 312 314 312
query28 7638 2027 2033 2027
query29 1139 607 593 593
query30 279 173 176 173
query31 951 742 755 742
query32 96 54 55 54
query33 761 261 261 261
query34 1014 478 473 473
query35 746 594 589 589
query36 1072 919 902 902
query37 181 96 72 72
query38 2944 2759 2788 2759
query39 854 783 795 783
query40 277 126 123 123
query41 47 45 45 45
query42 103 99 96 96
query43 577 546 519 519
query44 1203 718 731 718
query45 179 164 164 164
query46 1061 715 692 692
query47 1832 1772 1785 1772
query48 367 296 297 296
query49 1189 374 382 374
query50 763 379 391 379
query51 6880 6783 6763 6763
query52 107 87 97 87
query53 351 292 290 290
query54 980 428 418 418
query55 78 76 72 72
query56 262 275 239 239
query57 1122 1055 1032 1032
query58 229 206 210 206
query59 3297 3266 3104 3104
query60 278 261 256 256
query61 89 88 85 85
query62 654 447 464 447
query63 313 282 282 282
query64 9823 2288 1741 1741
query65 3183 3100 3133 3100
query66 1385 336 319 319
query67 15430 14727 14784 14727
query68 4623 530 527 527
query69 438 259 262 259
query70 1148 1108 1114 1108
query71 409 265 271 265
query72 7177 5601 2761 2761
query73 698 322 321 321
query74 5976 5599 5534 5534
query75 3424 2622 2618 2618
query76 2821 978 945 945
query77 475 268 268 268
query78 10117 9927 9901 9901
query79 2091 513 503 503
query80 937 439 434 434
query81 537 245 242 242
query82 646 97 93 93
query83 261 173 169 169
query84 250 86 83 83
query85 1888 328 269 269
query86 484 308 308 308
query87 3330 3155 3109 3109
query88 3937 2306 2305 2305
query89 466 408 396 396
query90 2007 193 189 189
query91 123 99 98 98
query92 58 47 49 47
query93 2408 511 498 498
query94 1233 189 185 185
query95 395 305 310 305
query96 585 262 262 262
query97 3201 3063 3055 3055
query98 246 220 210 210
query99 1091 839 820 820
Total cold run time: 275630 ms
Total hot run time: 168772 ms
ClickBench: Total hot run time: 29.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d4e21fcc6d1fa35b1768c8402621513e7209e849, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.03 0.04
query3 0.23 0.05 0.04
query4 1.67 0.07 0.10
query5 0.50 0.51 0.50
query6 1.13 0.71 0.72
query7 0.02 0.02 0.02
query8 0.05 0.04 0.05
query9 0.54 0.48 0.50
query10 0.56 0.55 0.55
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.59 0.58 0.60
query14 0.77 0.77 0.78
query15 0.83 0.81 0.82
query16 0.36 0.35 0.37
query17 0.98 0.98 0.99
query18 0.22 0.24 0.23
query19 1.74 1.82 1.80
query20 0.02 0.01 0.01
query21 15.53 0.66 0.64
query22 4.67 7.50 1.34
query23 18.28 1.35 1.20
query24 1.38 0.33 0.21
query25 0.14 0.08 0.08
query26 0.25 0.16 0.16
query27 0.09 0.09 0.08
query28 13.50 1.02 0.99
query29 13.16 3.30 3.28
query30 0.25 0.07 0.05
query31 2.87 0.38 0.37
query32 3.29 0.47 0.48
query33 2.89 2.90 2.91
query34 16.91 4.42 4.40
query35 4.50 4.50 4.46
query36 0.64 0.45 0.46
query37 0.17 0.16 0.15
query38 0.15 0.14 0.14
query39 0.05 0.04 0.03
query40 0.19 0.14 0.14
query41 0.09 0.05 0.04
query42 0.05 0.05 0.04
query43 0.04 0.04 0.04
Total cold run time: 109.72 s
Total hot run time: 29.89 s
PR approved by at least one committer and no changes requested.