doris
doris copied to clipboard
Published
20 hours ago •
apache
apache
[Opt](Iceberg) Simplify the code of getting time travel snapshotId
Proposed changes
Optimize the code of getting snapshotId. We can use the Apache Iceberg funtion to get time travel snapshotId.
Issue Number: close #xxx
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
TPC-DS: Total hot run time: 186215 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 522b320425f3ee577890636d7c2bc0c545f1af8b, data reload: false
query1 909 353 368 353
query2 6303 2288 2189 2189
query3 6661 214 205 205
query4 23096 21787 21926 21787
query5 3811 422 434 422
query6 278 197 160 160
query7 4556 285 296 285
query8 235 168 189 168
query9 8450 2314 2333 2314
query10 416 241 260 241
query11 15158 14868 14821 14821
query12 118 89 82 82
query13 1625 371 364 364
query14 10367 6554 7411 6554
query15 243 166 161 161
query16 8101 244 246 244
query17 1733 536 543 536
query18 2051 262 259 259
query19 198 148 141 141
query20 86 80 82 80
query21 184 146 124 124
query22 4990 4820 4795 4795
query23 33747 33066 33150 33066
query24 10582 2904 2861 2861
query25 587 373 361 361
query26 1120 147 142 142
query27 2813 308 304 304
query28 7552 1963 1943 1943
query29 842 584 581 581
query30 232 147 151 147
query31 961 708 704 704
query32 86 49 50 49
query33 733 234 243 234
query34 1064 462 456 456
query35 813 664 643 643
query36 1073 939 910 910
query37 123 63 63 63
query38 3109 3018 3051 3018
query39 1575 1616 1532 1532
query40 207 119 121 119
query41 40 37 37 37
query42 101 90 99 90
query43 565 521 537 521
query44 1201 719 724 719
query45 279 266 259 259
query46 1070 707 695 695
query47 1933 1840 1844 1840
query48 373 293 296 293
query49 882 384 393 384
query50 783 373 375 373
query51 6868 6699 6674 6674
query52 104 88 99 88
query53 346 277 281 277
query54 301 240 237 237
query55 76 71 72 71
query56 242 230 222 222
query57 1190 1134 1127 1127
query58 223 208 204 204
query59 3266 3099 3085 3085
query60 255 231 234 231
query61 108 105 107 105
query62 644 451 431 431
query63 300 273 272 272
query64 8590 7264 7208 7208
query65 3128 3062 3037 3037
query66 1370 335 422 335
query67 15750 14929 15009 14929
query68 9477 545 548 545
query69 596 305 295 295
query70 1322 1031 1117 1031
query71 504 255 277 255
query72 8329 2557 2408 2408
query73 1539 317 309 309
query74 6434 6068 6026 6026
query75 4586 2631 2666 2631
query76 5722 946 937 937
query77 662 258 256 256
query78 11112 10157 10360 10157
query79 10430 507 506 506
query80 1413 419 417 417
query81 503 221 216 216
query82 239 84 93 84
query83 224 163 163 163
query84 260 82 82 82
query85 1150 264 297 264
query86 440 316 319 316
query87 3304 3075 3073 3073
query88 5107 2384 2384 2384
query89 499 361 366 361
query90 2054 178 176 176
query91 119 94 96 94
query92 58 47 46 46
query93 6744 492 489 489
query94 1286 177 175 175
query95 1090 1097 1086 1086
query96 604 263 261 261
query97 3113 2922 2970 2922
query98 233 217 221 217
query99 1213 898 871 871
Total cold run time: 307142 ms
Total hot run time: 186215 ms
PR approved by at least one committer and no changes requested.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
TPC-H: Total hot run time: 40507 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 123f755ca3aa8e98a542981ab5ed96aa3236179e, data reload: false
------ Round 1 ----------------------------------
q1 17678 4451 4294 4294
q2 2026 196 200 196
q3 10516 1229 1016 1016
q4 10206 828 802 802
q5 7501 2710 2681 2681
q6 225 138 143 138
q7 972 595 600 595
q8 9237 2091 2074 2074
q9 8940 6541 6478 6478
q10 9025 3780 3763 3763
q11 469 239 233 233
q12 461 236 239 236
q13 18721 2975 2983 2975
q14 274 218 220 218
q15 534 476 483 476
q16 506 383 375 375
q17 989 603 730 603
q18 8144 7457 7371 7371
q19 7754 1512 1369 1369
q20 664 332 339 332
q21 4947 3972 3942 3942
q22 402 342 340 340
Total cold run time: 120191 ms
Total hot run time: 40507 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4416 4242 4224 4224
q2 370 263 253 253
q3 3192 2939 2918 2918
q4 2074 1756 1691 1691
q5 5481 5487 5502 5487
q6 227 143 140 140
q7 2267 1833 1846 1833
q8 3282 3467 3409 3409
q9 8743 8837 8742 8742
q10 4203 3747 3850 3747
q11 609 510 497 497
q12 851 663 652 652
q13 16266 3202 3210 3202
q14 311 265 286 265
q15 541 500 491 491
q16 495 449 460 449
q17 1818 1508 1490 1490
q18 8175 8060 7822 7822
q19 1816 1530 1496 1496
q20 2155 1876 1875 1875
q21 5300 4802 4936 4802
q22 648 558 569 558
Total cold run time: 73240 ms
Total hot run time: 56043 ms
TPC-DS: Total hot run time: 174978 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 123f755ca3aa8e98a542981ab5ed96aa3236179e, data reload: false
query1 903 398 376 376
query2 6436 2386 2351 2351
query3 6629 205 220 205
query4 21249 17363 17305 17305
query5 3770 502 486 486
query6 267 173 164 164
query7 4590 296 297 296
query8 303 308 280 280
query9 8671 2439 2389 2389
query10 569 324 289 289
query11 10434 10070 10011 10011
query12 125 87 85 85
query13 1669 374 374 374
query14 9456 7522 7671 7522
query15 233 193 185 185
query16 7896 285 269 269
query17 1876 567 539 539
query18 2009 286 284 284
query19 217 159 164 159
query20 94 84 84 84
query21 207 142 129 129
query22 4300 3991 4063 3991
query23 33824 33862 33618 33618
query24 10651 2908 2971 2908
query25 612 405 396 396
query26 727 159 163 159
query27 2262 333 333 333
query28 5945 2155 2174 2155
query29 903 678 640 640
query30 273 160 156 156
query31 1011 784 789 784
query32 105 58 57 57
query33 697 330 314 314
query34 890 505 505 505
query35 804 673 665 665
query36 1150 989 971 971
query37 145 82 84 82
query38 2931 2819 2878 2819
query39 928 846 843 843
query40 222 146 137 137
query41 58 58 58 58
query42 117 103 106 103
query43 608 575 537 537
query44 1077 832 742 742
query45 186 168 167 167
query46 1083 724 701 701
query47 1844 1772 1797 1772
query48 382 295 299 295
query49 877 412 420 412
query50 769 387 395 387
query51 6877 6864 6720 6720
query52 108 91 96 91
query53 364 307 292 292
query54 916 443 440 440
query55 74 74 74 74
query56 299 274 273 273
query57 1132 1076 1071 1071
query58 251 242 265 242
query59 3463 3346 3138 3138
query60 338 286 291 286
query61 100 114 89 89
query62 600 463 444 444
query63 320 304 299 299
query64 8575 2273 1729 1729
query65 3200 3108 3124 3108
query66 753 334 362 334
query67 15569 14911 14885 14885
query68 7883 541 552 541
query69 740 426 329 329
query70 1220 1121 1128 1121
query71 517 291 286 286
query72 9052 5742 5616 5616
query73 820 328 321 321
query74 5862 5571 5515 5515
query75 5313 2723 2669 2669
query76 4949 908 928 908
query77 796 307 313 307
query78 10466 9819 9781 9781
query79 7864 521 512 512
query80 1106 479 469 469
query81 557 226 220 220
query82 775 108 109 108
query83 328 170 165 165
query84 276 87 87 87
query85 1282 283 268 268
query86 432 329 310 310
query87 3387 3161 3072 3072
query88 4957 2380 2370 2370
query89 489 388 397 388
query90 2011 185 190 185
query91 129 99 99 99
query92 62 51 47 47
query93 6253 506 508 506
query94 1237 188 187 187
query95 403 303 308 303
query96 599 261 264 261
query97 3167 3009 2974 2974
query98 224 212 196 196
query99 1244 853 874 853
Total cold run time: 290323 ms
Total hot run time: 174978 ms
ClickBench: Total hot run time: 31.42 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 123f755ca3aa8e98a542981ab5ed96aa3236179e, data reload: false
query1 0.04 0.04 0.03
query2 0.07 0.04 0.04
query3 0.23 0.05 0.05
query4 1.68 0.08 0.07
query5 0.48 0.48 0.49
query6 1.16 0.72 0.74
query7 0.02 0.01 0.01
query8 0.05 0.05 0.05
query9 0.56 0.51 0.50
query10 0.54 0.55 0.54
query11 0.15 0.11 0.12
query12 0.14 0.12 0.12
query13 0.60 0.59 0.60
query14 0.76 0.78 0.77
query15 0.85 0.80 0.82
query16 0.38 0.38 0.37
query17 1.01 1.02 1.03
query18 0.26 0.22 0.27
query19 1.81 1.77 1.78
query20 0.01 0.01 0.01
query21 15.51 0.72 0.66
query22 3.82 6.99 2.55
query23 18.25 1.37 1.26
query24 2.14 0.23 0.22
query25 0.16 0.09 0.09
query26 0.26 0.18 0.18
query27 0.08 0.08 0.08
query28 13.20 1.02 1.00
query29 12.61 3.35 3.30
query30 0.26 0.06 0.05
query31 3.00 0.40 0.41
query32 3.24 0.48 0.48
query33 2.86 2.90 2.91
query34 17.16 4.50 4.40
query35 4.49 4.51 4.52
query36 0.66 0.47 0.51
query37 0.19 0.16 0.16
query38 0.15 0.14 0.14
query39 0.04 0.03 0.03
query40 0.18 0.14 0.14
query41 0.10 0.05 0.05
query42 0.05 0.05 0.04
query43 0.04 0.04 0.04
Total cold run time: 109.25 s
Total hot run time: 31.42 s