doris
doris copied to clipboard
Published
20 hours ago •
apache
apache
[feat](nereids) support Iceberg time travel syntax
Proposed changes
https://github.com/apache/doris/pull/15418 added Iceberg time travel in legacy parser but not added this syntax Neredis. If we enable nereids and disable fallback to original palnner, time travel won't be available.
This PR added time travel syntas in Neredis.
BTW, we already have nereids time travel regression-test in https://github.com/apache/doris/blob/master/regression-test/suites/external_table_p2/iceberg/test_external_catalog_icebergv2_nereids.groovy, this regression-test will always fail without this PR. https://github.com/apache/doris/blob/88530bf9437e60190f198252ab82f53fa53d4c10/regression-test/suites/external_table_p2/iceberg/test_external_catalog_icebergv2_nereids.groovy#L70-L75
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
TPC-H: Total hot run time: 39892 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa83cce7bede1c1f53806f37981ee6ab5f4bd6b7, data reload: false
------ Round 1 ----------------------------------
q1 17764 4608 4366 4366
q2 2618 188 193 188
q3 10985 1200 1167 1167
q4 10630 807 746 746
q5 7638 2661 2704 2661
q6 225 134 132 132
q7 944 528 515 515
q8 9204 2063 2021 2021
q9 8583 6516 6516 6516
q10 8883 3776 3659 3659
q11 442 239 239 239
q12 462 216 213 213
q13 17754 2932 2941 2932
q14 253 223 220 220
q15 503 477 471 471
q16 517 384 380 380
q17 958 733 817 733
q18 8020 7543 7459 7459
q19 4600 1517 1518 1517
q20 641 338 311 311
q21 5083 3168 3267 3168
q22 348 278 279 278
Total cold run time: 117055 ms
Total hot run time: 39892 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4371 4242 4243 4242
q2 375 271 261 261
q3 2976 2798 2694 2694
q4 1871 1589 1541 1541
q5 5285 5254 5252 5252
q6 216 124 125 124
q7 1736 1377 1345 1345
q8 3168 3315 3310 3310
q9 8262 8344 8325 8325
q10 3834 3620 3638 3620
q11 582 474 486 474
q12 772 593 600 593
q13 17639 2987 2997 2987
q14 287 265 267 265
q15 527 468 463 463
q16 459 400 405 400
q17 1757 1475 1469 1469
q18 7546 7566 7406 7406
q19 1638 1567 1546 1546
q20 1945 1752 1785 1752
q21 4833 4933 4830 4830
q22 563 501 494 494
Total cold run time: 70642 ms
Total hot run time: 53393 ms
TPC-DS: Total hot run time: 186552 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa83cce7bede1c1f53806f37981ee6ab5f4bd6b7, data reload: false
query1 918 370 341 341
query2 6445 2382 2385 2382
query3 6655 207 210 207
query4 23063 21131 21075 21075
query5 4155 421 423 421
query6 267 173 170 170
query7 4592 290 290 290
query8 258 189 187 187
query9 8738 2404 2392 2392
query10 444 250 257 250
query11 14772 14110 14330 14110
query12 133 94 91 91
query13 1649 371 366 366
query14 10595 8380 7599 7599
query15 262 171 172 171
query16 8174 257 263 257
query17 1867 563 542 542
query18 2101 273 280 273
query19 328 147 147 147
query20 90 89 88 88
query21 212 127 124 124
query22 5064 4883 4805 4805
query23 34218 33801 33493 33493
query24 11594 3005 2945 2945
query25 635 380 358 358
query26 1736 153 152 152
query27 2815 317 322 317
query28 7264 2053 2030 2030
query29 1004 603 591 591
query30 291 149 150 149
query31 950 745 728 728
query32 92 52 55 52
query33 751 249 246 246
query34 1022 485 493 485
query35 813 683 672 672
query36 1066 913 912 912
query37 175 65 65 65
query38 2881 2737 2731 2731
query39 1631 1569 1538 1538
query40 282 125 124 124
query41 41 40 39 39
query42 102 97 100 97
query43 584 515 542 515
query44 1201 714 739 714
query45 260 250 259 250
query46 1071 725 704 704
query47 1964 1902 1896 1896
query48 371 297 291 291
query49 1184 393 396 393
query50 785 386 392 386
query51 6895 6765 6696 6696
query52 104 83 90 83
query53 348 276 274 274
query54 943 431 429 429
query55 73 71 72 71
query56 238 223 217 217
query57 1244 1124 1139 1124
query58 224 194 197 194
query59 3378 3158 3130 3130
query60 263 238 238 238
query61 90 88 86 86
query62 668 476 473 473
query63 309 279 277 277
query64 9766 7366 7325 7325
query65 3146 3076 3076 3076
query66 1382 349 344 344
query67 15627 14958 15130 14958
query68 4906 538 533 533
query69 525 300 306 300
query70 1154 1097 1118 1097
query71 427 267 264 264
query72 7551 2591 2348 2348
query73 724 322 324 322
query74 6656 6142 5958 5958
query75 3630 2601 2602 2601
query76 3610 986 985 985
query77 639 276 263 263
query78 10574 10090 10076 10076
query79 1349 519 508 508
query80 1062 432 436 432
query81 493 220 220 220
query82 653 95 94 94
query83 193 163 167 163
query84 263 93 89 89
query85 1511 264 258 258
query86 403 289 305 289
query87 3334 3128 3119 3119
query88 3452 2424 2410 2410
query89 474 389 383 383
query90 2009 188 190 188
query91 121 98 97 97
query92 58 48 48 48
query93 1285 510 504 504
query94 1120 179 188 179
query95 384 304 309 304
query96 584 268 271 268
query97 3168 2993 3014 2993
query98 245 220 214 214
query99 1302 915 889 889
Total cold run time: 286122 ms
Total hot run time: 186552 ms
PR approved by at least one committer and no changes requested.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
TPC-H: Total hot run time: 40423 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f18eb36bd4f1121530cf1b910b13b85645a36674, data reload: false
------ Round 1 ----------------------------------
q1 17675 4338 4184 4184
q2 2016 196 190 190
q3 10575 1292 1218 1218
q4 10547 760 777 760
q5 7523 2668 2632 2632
q6 211 132 132 132
q7 982 585 594 585
q8 9634 2091 2065 2065
q9 9207 6646 6597 6597
q10 9601 3717 3855 3717
q11 444 241 241 241
q12 447 214 216 214
q13 17778 2946 2956 2946
q14 269 207 230 207
q15 514 466 477 466
q16 522 386 384 384
q17 968 663 730 663
q18 8108 7325 7419 7325
q19 4221 1541 1494 1494
q20 652 305 302 302
q21 5318 3824 3867 3824
q22 349 277 290 277
Total cold run time: 117561 ms
Total hot run time: 40423 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4303 4200 4195 4195
q2 375 276 273 273
q3 2988 2761 2716 2716
q4 1823 1521 1594 1521
q5 5238 5251 5257 5251
q6 210 125 123 123
q7 2228 1867 1893 1867
q8 3158 3280 3328 3280
q9 8295 8285 8293 8285
q10 3819 3669 3650 3650
q11 580 499 494 494
q12 768 573 588 573
q13 16386 2960 2976 2960
q14 305 276 274 274
q15 516 478 471 471
q16 451 415 421 415
q17 1757 1467 1477 1467
q18 7710 7609 7331 7331
q19 1645 1593 1526 1526
q20 1953 1778 1761 1761
q21 4787 4766 4774 4766
q22 551 485 475 475
Total cold run time: 69846 ms
Total hot run time: 53674 ms
TPC-DS: Total hot run time: 187388 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f18eb36bd4f1121530cf1b910b13b85645a36674, data reload: false
query1 915 365 345 345
query2 6454 2395 2333 2333
query3 6638 208 208 208
query4 23075 21386 21203 21203
query5 4146 421 422 421
query6 256 172 177 172
query7 4579 296 283 283
query8 243 191 189 189
query9 8480 2491 2450 2450
query10 459 242 251 242
query11 14704 14285 14240 14240
query12 137 91 89 89
query13 1634 382 379 379
query14 10510 8194 7641 7641
query15 228 167 170 167
query16 8080 267 260 260
query17 1854 580 539 539
query18 2059 267 269 267
query19 214 155 151 151
query20 109 87 89 87
query21 202 130 134 130
query22 5027 4828 4887 4828
query23 34147 33526 33475 33475
query24 11904 2942 2943 2942
query25 624 389 362 362
query26 1695 155 153 153
query27 2987 322 336 322
query28 7441 2072 2061 2061
query29 982 626 611 611
query30 285 150 153 150
query31 975 766 742 742
query32 92 52 58 52
query33 751 249 238 238
query34 1052 517 499 499
query35 796 669 681 669
query36 1087 896 920 896
query37 133 67 67 67
query38 2890 2781 2752 2752
query39 1653 1556 1557 1556
query40 276 126 125 125
query41 41 39 39 39
query42 106 97 97 97
query43 571 549 572 549
query44 1218 733 737 733
query45 261 258 259 258
query46 1085 716 727 716
query47 1959 1893 1938 1893
query48 370 307 305 305
query49 1128 392 394 392
query50 777 392 399 392
query51 6830 6796 6854 6796
query52 107 86 92 86
query53 350 287 284 284
query54 965 431 439 431
query55 80 77 81 77
query56 254 232 265 232
query57 1268 1196 1162 1162
query58 219 203 191 191
query59 3511 3218 3616 3218
query60 263 230 239 230
query61 92 87 85 85
query62 665 471 461 461
query63 309 295 286 286
query64 9793 7396 7356 7356
query65 3154 3096 3087 3087
query66 1383 333 347 333
query67 15474 14923 14864 14864
query68 4555 535 532 532
query69 482 299 299 299
query70 1156 1138 1076 1076
query71 405 270 260 260
query72 7401 2675 2353 2353
query73 698 326 329 326
query74 6544 6200 6154 6154
query75 3344 2613 2622 2613
query76 2611 991 979 979
query77 402 269 264 264
query78 10570 10224 10001 10001
query79 2472 517 519 517
query80 1041 482 426 426
query81 508 223 222 222
query82 743 91 94 91
query83 247 164 173 164
query84 234 92 87 87
query85 1589 271 268 268
query86 521 304 310 304
query87 3312 3110 3108 3108
query88 4323 2447 2455 2447
query89 471 379 377 377
query90 1959 190 190 190
query91 127 97 98 97
query92 65 48 51 48
query93 1710 511 495 495
query94 1192 186 185 185
query95 415 303 307 303
query96 585 275 277 275
query97 3158 2970 2963 2963
query98 224 220 212 212
query99 1139 903 918 903
Total cold run time: 286209 ms
Total hot run time: 187388 ms
TPC-DS: Total hot run time: 186741 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f18eb36bd4f1121530cf1b910b13b85645a36674, data reload: false
query1 911 388 374 374
query2 6321 2456 2329 2329
query3 6654 218 216 216
query4 26214 21774 21931 21774
query5 3789 426 429 426
query6 280 187 171 171
query7 4542 297 296 296
query8 239 186 192 186
query9 8596 2405 2463 2405
query10 426 246 261 246
query11 15191 14946 14840 14840
query12 119 94 85 85
query13 1569 356 372 356
query14 10273 6770 7665 6770
query15 236 168 166 166
query16 8115 255 255 255
query17 1706 549 526 526
query18 2085 265 267 265
query19 194 151 142 142
query20 88 81 80 80
query21 186 130 129 129
query22 5099 4946 4918 4918
query23 33926 33701 33356 33356
query24 10680 2841 2898 2841
query25 597 360 355 355
query26 1169 152 147 147
query27 3006 310 322 310
query28 7136 2006 1995 1995
query29 853 635 607 607
query30 262 171 169 169
query31 987 739 748 739
query32 94 49 57 49
query33 734 245 238 238
query34 1069 490 472 472
query35 774 666 651 651
query36 1045 917 922 917
query37 138 70 68 68
query38 2903 2764 2835 2764
query39 1595 1567 1558 1558
query40 192 125 119 119
query41 47 42 41 41
query42 100 96 93 93
query43 548 551 527 527
query44 1186 725 732 725
query45 266 248 258 248
query46 1068 722 700 700
query47 1981 1889 1879 1879
query48 365 305 292 292
query49 895 386 381 381
query50 749 376 385 376
query51 6857 6789 6807 6789
query52 98 94 92 92
query53 346 280 278 278
query54 841 416 420 416
query55 75 72 72 72
query56 236 215 221 215
query57 1209 1178 1122 1122
query58 214 194 196 194
query59 3368 3181 2952 2952
query60 279 230 226 226
query61 86 85 88 85
query62 637 472 454 454
query63 311 280 283 280
query64 8802 7369 7396 7369
query65 3096 3076 3055 3055
query66 804 325 334 325
query67 15816 15012 14865 14865
query68 4531 517 520 517
query69 477 300 299 299
query70 1134 1085 1091 1085
query71 423 266 261 261
query72 7095 2591 2324 2324
query73 719 317 319 317
query74 6611 6195 6221 6195
query75 3415 2641 2607 2607
query76 2702 955 975 955
query77 365 261 263 261
query78 10568 10201 10017 10017
query79 2521 515 508 508
query80 1052 425 417 417
query81 579 240 241 240
query82 943 94 95 94
query83 260 164 163 163
query84 232 87 84 84
query85 1179 264 255 255
query86 473 318 295 295
query87 3352 3118 3201 3118
query88 4189 2300 2317 2300
query89 467 384 390 384
query90 1938 193 185 185
query91 123 97 107 97
query92 61 48 48 48
query93 1810 484 484 484
query94 1145 173 172 172
query95 387 293 315 293
query96 586 260 268 260
query97 3165 3066 3014 3014
query98 237 227 215 215
query99 1177 916 934 916
Total cold run time: 284189 ms
Total hot run time: 186741 ms
ClickBench: Total hot run time: 30.16 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f18eb36bd4f1121530cf1b910b13b85645a36674, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.24 0.04 0.05
query4 1.66 0.07 0.06
query5 0.49 0.49 0.52
query6 1.12 0.73 0.72
query7 0.01 0.02 0.01
query8 0.06 0.04 0.04
query9 0.53 0.48 0.48
query10 0.54 0.52 0.53
query11 0.16 0.12 0.11
query12 0.15 0.12 0.11
query13 0.59 0.59 0.60
query14 0.77 0.77 0.79
query15 0.82 0.80 0.80
query16 0.35 0.37 0.36
query17 0.97 0.93 0.92
query18 0.22 0.22 0.26
query19 1.79 1.69 1.70
query20 0.02 0.01 0.01
query21 15.73 0.65 0.64
query22 4.23 7.67 1.73
query23 18.31 1.45 1.29
query24 1.29 0.45 0.21
query25 0.14 0.09 0.08
query26 0.27 0.17 0.18
query27 0.08 0.08 0.08
query28 13.31 1.01 1.00
query29 13.14 3.38 3.33
query30 0.24 0.06 0.06
query31 2.86 0.39 0.37
query32 3.28 0.47 0.47
query33 2.81 2.80 2.81
query34 17.04 4.39 4.47
query35 4.48 4.48 4.47
query36 0.71 0.49 0.48
query37 0.17 0.15 0.15
query38 0.15 0.14 0.14
query39 0.04 0.04 0.03
query40 0.16 0.14 0.14
query41 0.09 0.05 0.04
query42 0.06 0.04 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.24 s
Total hot run time: 30.16 s
@morrySnow Gentle ping. Can we merge this change? Or Is there anything else that needs to be done?
@morrySnow Gentle ping. Can we merge this change? Or Is there anything else that needs to be done?
we need 2 approve to merge pr. let's find another reviewer to review this PR
TPC-H: Total hot run time: 42135 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 761c29f4fd17b4c6f3d87d65c447d6cfefa67621, data reload: false
------ Round 1 ----------------------------------
q1 17595 4906 4260 4260
q2 2017 193 197 193
q3 10447 1187 1260 1187
q4 10224 913 866 866
q5 7499 2705 2741 2705
q6 218 136 134 134
q7 1062 598 588 588
q8 9227 2163 2135 2135
q9 9112 6784 6661 6661
q10 9948 3846 3941 3846
q11 438 245 263 245
q12 431 228 217 217
q13 17486 3213 3259 3213
q14 256 215 209 209
q15 535 465 486 465
q16 522 411 394 394
q17 1002 689 761 689
q18 8504 7983 7888 7888
q19 4204 1542 1577 1542
q20 650 319 333 319
q21 5309 4097 4141 4097
q22 369 282 298 282
Total cold run time: 117055 ms
Total hot run time: 42135 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4543 4397 4435 4397
q2 378 276 268 268
q3 3168 2933 2755 2755
q4 1927 1607 1658 1607
q5 5529 5501 5572 5501
q6 221 124 125 124
q7 2354 1956 1964 1956
q8 3268 3443 3421 3421
q9 8705 8698 8620 8620
q10 3946 3863 3852 3852
q11 612 502 495 495
q12 818 632 616 616
q13 17417 3181 3086 3086
q14 285 272 257 257
q15 509 475 463 463
q16 470 409 413 409
q17 1778 1490 1457 1457
q18 7678 7648 7395 7395
q19 1692 1532 1605 1532
q20 2055 1814 1776 1776
q21 10752 4985 4855 4855
q22 598 503 490 490
Total cold run time: 78703 ms
Total hot run time: 55332 ms
TPC-DS: Total hot run time: 181302 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 761c29f4fd17b4c6f3d87d65c447d6cfefa67621, data reload: false
query1 910 397 374 374
query2 6440 2521 2392 2392
query3 6650 211 203 203
query4 22935 21234 21131 21131
query5 4107 422 426 422
query6 259 172 172 172
query7 4586 300 295 295
query8 235 191 188 188
query9 8479 2390 2380 2380
query10 452 251 258 251
query11 14855 14201 14072 14072
query12 135 94 88 88
query13 1635 390 359 359
query14 10454 8648 7741 7741
query15 248 172 165 165
query16 8071 271 251 251
query17 1718 541 542 541
query18 2100 281 265 265
query19 189 144 150 144
query20 95 86 86 86
query21 187 129 126 126
query22 5062 4847 4765 4765
query23 34198 33565 33268 33268
query24 6699 2779 2858 2779
query25 541 375 359 359
query26 700 154 156 154
query27 1887 311 330 311
query28 3619 2061 2048 2048
query29 879 610 595 595
query30 256 182 173 173
query31 978 775 727 727
query32 91 52 53 52
query33 485 239 246 239
query34 871 479 492 479
query35 750 678 659 659
query36 1062 908 899 899
query37 109 70 73 70
query38 2906 2773 2775 2773
query39 1622 1531 1618 1531
query40 211 130 133 130
query41 46 44 45 44
query42 104 96 96 96
query43 558 551 550 550
query44 1082 728 753 728
query45 263 266 237 237
query46 1060 730 712 712
query47 1973 1862 1872 1862
query48 391 304 288 288
query49 786 393 406 393
query50 773 391 387 387
query51 6835 6691 6753 6691
query52 105 91 91 91
query53 346 293 285 285
query54 526 429 430 429
query55 75 73 73 73
query56 238 219 224 219
query57 1230 1143 1147 1143
query58 220 199 199 199
query59 3523 3190 3225 3190
query60 255 237 239 237
query61 90 129 88 88
query62 614 457 467 457
query63 308 277 285 277
query64 8453 2245 1743 1743
query65 3204 3082 3147 3082
query66 789 333 340 333
query67 15485 15202 15279 15202
query68 4566 523 535 523
query69 481 313 312 312
query70 1161 1133 1166 1133
query71 390 282 274 274
query72 7887 2605 2350 2350
query73 713 316 316 316
query74 6704 6190 6215 6190
query75 3318 2630 2604 2604
query76 2328 938 1007 938
query77 396 268 268 268
query78 10567 10172 10107 10107
query79 1670 508 532 508
query80 966 433 444 433
query81 526 249 243 243
query82 961 102 101 101
query83 254 171 172 171
query84 256 89 94 89
query85 1049 334 262 262
query86 465 309 297 297
query87 3283 3065 3080 3065
query88 3061 2324 2333 2324
query89 475 379 388 379
query90 1976 201 193 193
query91 126 97 100 97
query92 58 50 48 48
query93 1470 505 495 495
query94 1094 189 184 184
query95 400 306 304 304
query96 577 263 269 263
query97 3178 3023 3054 3023
query98 240 229 217 217
query99 1230 898 937 898
Total cold run time: 268824 ms
Total hot run time: 181302 ms
ClickBench: Total hot run time: 30.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 761c29f4fd17b4c6f3d87d65c447d6cfefa67621, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.24 0.06 0.05
query4 1.66 0.08 0.08
query5 0.50 0.50 0.51
query6 1.12 0.72 0.73
query7 0.02 0.01 0.01
query8 0.05 0.04 0.05
query9 0.55 0.49 0.49
query10 0.54 0.55 0.53
query11 0.15 0.12 0.11
query12 0.16 0.12 0.11
query13 0.60 0.60 0.60
query14 0.78 0.78 0.77
query15 0.85 0.82 0.81
query16 0.38 0.36 0.37
query17 1.01 0.99 0.99
query18 0.21 0.26 0.21
query19 1.80 1.67 1.68
query20 0.02 0.01 0.01
query21 15.73 0.65 0.64
query22 4.50 6.32 2.25
query23 18.27 1.43 1.35
query24 1.77 0.23 0.26
query25 0.15 0.08 0.08
query26 0.26 0.17 0.17
query27 0.08 0.08 0.08
query28 13.33 1.03 0.99
query29 13.76 3.27 3.22
query30 0.24 0.07 0.06
query31 2.83 0.41 0.39
query32 3.28 0.46 0.47
query33 2.91 2.95 2.95
query34 17.24 4.35 4.39
query35 4.47 4.50 4.63
query36 0.66 0.49 0.45
query37 0.17 0.17 0.15
query38 0.15 0.15 0.14
query39 0.04 0.04 0.04
query40 0.16 0.15 0.14
query41 0.09 0.05 0.04
query42 0.05 0.05 0.04
query43 0.04 0.04 0.04
Total cold run time: 110.94 s
Total hot run time: 30.89 s
PR approved by at least one committer and no changes requested.