doris
doris copied to clipboard
apache
[func](jsonb)support json_depth function
Proposed changes
Issue Number: close #xxx
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
TeamCity be ut coverage result: Function Coverage: 36.32% (8022/22086) Line Coverage: 28.63% (64423/225057) Region Coverage: 27.53% (33455/121534) Branch Coverage: 24.19% (17120/70760) Coverage Report: http://coverage.selectdb-in.cc/coverage/dd2887e3beadd6308d38c52b832db93edb468bb5_dd2887e3beadd6308d38c52b832db93edb468bb5/report/index.html
(From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.77 seconds stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s storage size: 17162240464 Bytes
(From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.67 seconds stream load tsv: 550 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s storage size: 17095806970 Bytes
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 8c7a97eaa29c6d3524a4ed1749616234d1c39653, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5314 5072 5257 5072
q2 358 148 159 148
q3 2022 2020 1999 1999
q4 1388 1370 1351 1351
q5 3963 3933 3939 3933
q6 253 129 131 129
q7 1454 878 891 878
q8 2777 2765 2758 2758
q9 9818 9541 9664 9541
q10 3451 3527 3525 3525
q11 388 250 247 247
q12 424 278 282 278
q13 4588 4202 4162 4162
q14 327 296 287 287
q15 625 571 530 530
q16 674 586 576 576
q17 1131 1103 1093 1093
q18 8108 7721 7653 7653
q19 1657 1690 1691 1690
q20 557 310 301 301
q21 4664 4332 4349 4332
q22 509 409 388 388
Total cold run time: 54450 ms
Total hot run time: 50871 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5012 5018 5025 5018
q2 331 218 235 218
q3 4063 4001 3995 3995
q4 2777 2755 2769 2755
q5 9728 9622 9651 9622
q6 246 122 119 119
q7 3038 2498 2511 2498
q8 4816 4815 4834 4815
q9 13300 13116 13135 13116
q10 4066 4144 4159 4144
q11 736 641 706 641
q12 973 861 831 831
q13 4264 3865 3809 3809
q14 386 347 343 343
q15 574 536 558 536
q16 755 693 661 661
q17 4053 4001 3917 3917
q18 9675 9484 9479 9479
q19 1767 1807 1768 1768
q20 2379 2058 2042 2042
q21 8979 8880 8888 8880
q22 988 902 828 828
Total cold run time: 82906 ms
Total hot run time: 80035 ms
TeamCity be ut coverage result: Function Coverage: 36.80% (8421/22886) Line Coverage: 29.27% (68407/233688) Region Coverage: 27.87% (35357/126862) Branch Coverage: 24.64% (18069/73340) Coverage Report: http://coverage.selectdb-in.cc/coverage/36fb0a1f3e17c0ef37293e889b409bde61b06c27_36fb0a1f3e17c0ef37293e889b409bde61b06c27/report/index.html
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 36fb0a1f3e17c0ef37293e889b409bde61b06c27, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5220 5043 5050 5043
q2 359 177 159 159
q3 2041 2058 2012 2012
q4 1401 1360 1341 1341
q5 3971 3988 3972 3972
q6 255 131 126 126
q7 1434 885 886 885
q8 2771 2789 2760 2760
q9 9768 9632 9580 9580
q10 3452 3544 3526 3526
q11 383 257 260 257
q12 426 286 283 283
q13 4588 4128 4198 4128
q14 317 286 290 286
q15 601 540 532 532
q16 671 590 577 577
q17 1132 1054 1063 1054
q18 8124 7704 7661 7661
q19 1675 1682 1678 1678
q20 561 294 311 294
q21 4648 4355 4357 4355
q22 508 402 409 402
Total cold run time: 54306 ms
Total hot run time: 50911 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4978 4998 4971 4971
q2 343 218 226 218
q3 4067 4007 4020 4007
q4 2770 2749 2743 2743
q5 9741 9678 9671 9671
q6 243 120 120 120
q7 3059 2491 2513 2491
q8 4835 4801 4795 4795
q9 13238 13224 13139 13139
q10 4067 4177 4167 4167
q11 758 634 690 634
q12 988 841 834 834
q13 4276 3895 3898 3895
q14 372 354 353 353
q15 615 542 558 542
q16 772 676 671 671
q17 3969 3915 3920 3915
q18 9534 9419 9367 9367
q19 1811 1772 1775 1772
q20 2401 2083 2045 2045
q21 8861 8764 8723 8723
q22 909 844 826 826
Total cold run time: 82607 ms
Total hot run time: 79899 ms
(From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.65 seconds stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s storage size: 17096271919 Bytes
TeamCity be ut coverage result: Function Coverage: 36.80% (8421/22886) Line Coverage: 29.28% (68416/233691) Region Coverage: 27.87% (35362/126863) Branch Coverage: 24.64% (18069/73340) Coverage Report: http://coverage.selectdb-in.cc/coverage/36fb0a1f3e17c0ef37293e889b409bde61b06c27_36fb0a1f3e17c0ef37293e889b409bde61b06c27/report/index.html
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 36fb0a1f3e17c0ef37293e889b409bde61b06c27, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5249 5149 5191 5149
q2 378 147 159 147
q3 2043 1970 2020 1970
q4 1424 1302 1278 1278
q5 3997 3996 3991 3991
q6 253 127 127 127
q7 1427 886 905 886
q8 2783 2826 2826 2826
q9 10082 9712 9982 9712
q10 3471 3535 3554 3535
q11 394 264 249 249
q12 438 278 280 278
q13 4617 3814 3803 3803
q14 310 289 275 275
q15 614 554 540 540
q16 681 597 583 583
q17 1160 982 958 958
q18 7937 7490 7581 7490
q19 1685 1693 1667 1667
q20 563 323 296 296
q21 4480 4059 4070 4059
q22 517 407 395 395
Total cold run time: 54503 ms
Total hot run time: 50214 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5125 5113 5123 5113
q2 319 210 250 210
q3 4047 4037 4031 4031
q4 2817 2778 2764 2764
q5 9696 9688 9720 9688
q6 244 124 126 124
q7 3047 2567 2488 2488
q8 4861 4884 4878 4878
q9 13301 13244 13204 13204
q10 4092 4190 4176 4176
q11 735 660 634 634
q12 1007 864 829 829
q13 4287 3592 3594 3592
q14 375 347 347 347
q15 632 549 551 549
q16 789 689 684 684
q17 3964 3830 3906 3830
q18 9684 9178 9070 9070
q19 1957 1795 1809 1795
q20 2425 2069 2063 2063
q21 8791 8575 8635 8575
q22 926 888 882 882
Total cold run time: 83121 ms
Total hot run time: 79526 ms
(From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.87 seconds stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.0 seconds inserted 10000000 Rows, about 357K ops/s storage size: 17096577020 Bytes
TeamCity be ut coverage result: Function Coverage: 36.79% (8419/22886) Line Coverage: 29.27% (68391/233691) Region Coverage: 27.86% (35342/126863) Branch Coverage: 24.63% (18064/73340) Coverage Report: http://coverage.selectdb-in.cc/coverage/36fb0a1f3e17c0ef37293e889b409bde61b06c27_36fb0a1f3e17c0ef37293e889b409bde61b06c27/report/index.html
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 36fb0a1f3e17c0ef37293e889b409bde61b06c27, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5328 5102 5139 5102
q2 360 153 159 153
q3 2046 1956 1959 1956
q4 1377 1284 1263 1263
q5 3960 3928 4018 3928
q6 249 130 139 130
q7 1401 889 889 889
q8 2752 2777 2754 2754
q9 13084 9737 9541 9541
q10 3453 3502 3523 3502
q11 383 252 241 241
q12 448 295 291 291
q13 4591 3828 3859 3828
q14 314 302 283 283
q15 617 563 567 563
q16 671 594 580 580
q17 1141 982 957 957
q18 7763 7368 7280 7280
q19 1688 1654 1661 1654
q20 570 326 306 306
q21 4475 4020 4023 4020
q22 515 407 425 407
Total cold run time: 57186 ms
Total hot run time: 49628 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5047 5020 5022 5020
q2 334 223 235 223
q3 4016 3997 3912 3912
q4 2788 2757 2772 2757
q5 9545 9500 9459 9459
q6 247 128 132 128
q7 2661 2300 2337 2300
q8 4834 4835 4853 4835
q9 13197 13075 13060 13060
q10 4084 4181 4207 4181
q11 745 671 661 661
q12 993 836 818 818
q13 4309 3608 3564 3564
q14 400 340 346 340
q15 630 566 568 566
q16 752 689 690 689
q17 3923 3911 3874 3874
q18 9557 8887 9052 8887
q19 1787 1794 1783 1783
q20 2401 2054 2064 2054
q21 8734 8418 8575 8418
q22 975 841 859 841
Total cold run time: 81959 ms
Total hot run time: 78370 ms
(From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.51 seconds stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s storage size: 17097134897 Bytes
TeamCity be ut coverage result: Function Coverage: 36.77% (8411/22875) Line Coverage: 29.27% (68368/233576) Region Coverage: 27.88% (35343/126769) Branch Coverage: 24.64% (18064/73326) Coverage Report: http://coverage.selectdb-in.cc/coverage/36fb0a1f3e17c0ef37293e889b409bde61b06c27_36fb0a1f3e17c0ef37293e889b409bde61b06c27/report/index.html
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 36fb0a1f3e17c0ef37293e889b409bde61b06c27, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4933 4703 4707 4703
q2 356 159 162 159
q3 2026 1925 1940 1925
q4 1376 1248 1240 1240
q5 3975 3946 4028 3946
q6 253 131 131 131
q7 1384 891 883 883
q8 2751 2775 2741 2741
q9 9754 9544 9528 9528
q10 3447 3529 3508 3508
q11 383 241 249 241
q12 441 294 296 294
q13 4579 3803 3827 3803
q14 327 292 302 292
q15 589 557 525 525
q16 664 583 575 575
q17 1133 964 926 926
q18 7845 7229 7372 7229
q19 1671 1673 1677 1673
q20 531 313 289 289
q21 4379 3930 3964 3930
q22 476 376 374 374
Total cold run time: 53273 ms
Total hot run time: 48915 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4613 4611 4580 4580
q2 343 231 248 231
q3 4015 3992 3988 3988
q4 2691 2677 2673 2673
q5 9595 9677 9718 9677
q6 248 120 124 120
q7 2627 2302 2285 2285
q8 4480 4473 4438 4438
q9 13224 13069 13125 13069
q10 4079 4157 4178 4157
q11 773 658 682 658
q12 988 828 820 820
q13 4296 3562 3613 3562
q14 380 338 361 338
q15 578 516 520 516
q16 731 677 666 666
q17 3781 3776 3899 3776
q18 9435 8914 9070 8914
q19 1816 1769 1774 1769
q20 2392 2076 2044 2044
q21 8871 8458 8623 8458
q22 890 822 794 794
Total cold run time: 80846 ms
Total hot run time: 77533 ms
(From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.26 seconds stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s storage size: 17098605320 Bytes
TPC-H: Total hot run time: 39287 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a443af893bf5763eb5b92e41622ea9ef1cb21e72, data reload: false
------ Round 1 ----------------------------------
q1 17942 4695 4358 4358
q2 2548 209 201 201
q3 10997 1241 1249 1241
q4 10907 864 898 864
q5 7691 2839 2742 2742
q6 224 142 139 139
q7 1066 637 629 629
q8 9569 2122 2106 2106
q9 7624 6602 6604 6602
q10 8512 3517 3569 3517
q11 431 226 236 226
q12 405 227 221 221
q13 17771 3007 2953 2953
q14 271 222 236 222
q15 514 497 493 493
q16 525 421 379 379
q17 982 725 637 637
q18 7485 6880 6788 6788
q19 4882 1555 1535 1535
q20 663 336 319 319
q21 3592 2810 2871 2810
q22 398 307 305 305
Total cold run time: 114999 ms
Total hot run time: 39287 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4319 4210 4176 4176
q2 374 271 277 271
q3 3027 2745 2813 2745
q4 1909 1580 1573 1573
q5 5354 5329 5310 5310
q6 210 125 125 125
q7 2275 1899 1902 1899
q8 3217 3396 3378 3378
q9 8636 8627 8677 8627
q10 3940 3675 3744 3675
q11 582 489 502 489
q12 785 614 591 591
q13 17551 2953 3032 2953
q14 289 270 291 270
q15 510 466 470 466
q16 487 431 428 428
q17 1815 1478 1482 1478
q18 7589 7553 7379 7379
q19 1656 1548 1543 1543
q20 1946 1771 1726 1726
q21 5008 4691 4803 4691
q22 549 441 474 441
Total cold run time: 72028 ms
Total hot run time: 54234 ms
TPC-DS: Total hot run time: 183973 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a443af893bf5763eb5b92e41622ea9ef1cb21e72, data reload: false
query1 919 388 365 365
query2 6510 2599 2466 2466
query3 6645 212 213 212
query4 25189 21496 21406 21406
query5 4173 406 410 406
query6 262 191 173 173
query7 4583 289 283 283
query8 238 178 181 178
query9 8495 2346 2348 2346
query10 595 252 248 248
query11 14789 14238 14252 14238
query12 136 84 84 84
query13 1634 361 361 361
query14 9405 7538 7821 7538
query15 267 181 177 177
query16 8146 260 253 253
query17 1916 581 549 549
query18 2091 271 264 264
query19 255 148 151 148
query20 92 86 82 82
query21 197 123 129 123
query22 5007 4768 4795 4768
query23 33760 33152 33248 33152
query24 11905 2960 2997 2960
query25 647 362 370 362
query26 1735 148 148 148
query27 2959 304 311 304
query28 7616 2035 1997 1997
query29 1003 622 583 583
query30 296 178 174 174
query31 948 744 744 744
query32 87 53 56 53
query33 744 242 241 241
query34 1028 478 470 470
query35 838 700 695 695
query36 1063 913 924 913
query37 157 68 69 68
query38 3360 3143 3180 3143
query39 1592 1530 1532 1530
query40 275 126 124 124
query41 46 43 44 43
query42 103 94 94 94
query43 582 552 550 550
query44 1196 717 722 717
query45 275 269 271 269
query46 1089 725 711 711
query47 1919 1862 1846 1846
query48 373 290 298 290
query49 1184 396 378 378
query50 768 375 388 375
query51 6786 6652 6613 6613
query52 100 92 87 87
query53 351 284 282 282
query54 339 233 232 232
query55 76 73 74 73
query56 254 233 232 232
query57 1228 1122 1136 1122
query58 229 200 200 200
query59 3303 3148 3219 3148
query60 257 234 248 234
query61 113 110 110 110
query62 647 439 437 437
query63 308 274 280 274
query64 6520 4123 3761 3761
query65 3122 3060 3066 3060
query66 1410 341 354 341
query67 15336 15057 15103 15057
query68 7138 543 537 537
query69 532 317 305 305
query70 1289 1156 1204 1156
query71 479 278 275 275
query72 6629 2779 2564 2564
query73 754 320 323 320
query74 6916 6376 6533 6376
query75 3806 2666 2627 2627
query76 4524 969 1027 969
query77 661 269 280 269
query78 11128 10337 10112 10112
query79 10399 526 522 522
query80 1508 461 449 449
query81 521 256 252 252
query82 842 97 95 95
query83 214 308 162 162
query84 296 84 83 83
query85 1459 262 251 251
query86 490 298 294 294
query87 3468 3249 3311 3249
query88 5151 2337 2324 2324
query89 542 363 368 363
query90 1977 180 179 179
query91 126 100 100 100
query92 71 50 47 47
query93 7544 515 504 504
query94 1194 182 179 179
query95 390 306 299 299
query96 629 265 260 260
query97 3121 2933 2941 2933
query98 241 223 211 211
query99 1267 842 844 842
Total cold run time: 304777 ms
Total hot run time: 183973 ms
ClickBench: Total hot run time: 30.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a443af893bf5763eb5b92e41622ea9ef1cb21e72, data reload: false
query1 0.04 0.04 0.03
query2 0.07 0.04 0.04
query3 0.23 0.05 0.04
query4 1.68 0.07 0.07
query5 0.49 0.49 0.49
query6 1.48 0.72 0.72
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.54 0.50 0.50
query10 0.54 0.55 0.56
query11 0.16 0.12 0.12
query12 0.15 0.12 0.12
query13 0.60 0.60 0.58
query14 0.75 0.79 0.79
query15 0.82 0.81 0.81
query16 0.39 0.38 0.37
query17 0.92 1.03 1.02
query18 0.22 0.24 0.22
query19 1.84 1.67 1.74
query20 0.02 0.01 0.01
query21 15.42 0.64 0.63
query22 4.40 6.50 1.86
query23 18.27 1.30 1.17
query24 1.78 0.29 0.20
query25 0.14 0.09 0.08
query26 0.26 0.16 0.16
query27 0.09 0.08 0.08
query28 13.30 1.01 0.98
query29 12.63 3.30 3.30
query30 0.26 0.07 0.06
query31 2.89 0.39 0.37
query32 3.26 0.46 0.46
query33 2.78 2.84 2.83
query34 17.14 4.45 4.41
query35 4.48 4.46 4.43
query36 0.66 0.46 0.48
query37 0.18 0.14 0.15
query38 0.15 0.14 0.15
query39 0.05 0.04 0.04
query40 0.18 0.15 0.14
query41 0.10 0.04 0.05
query42 0.06 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.53 s
Total hot run time: 30.27 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit a443af893bf5763eb5b92e41622ea9ef1cb21e72 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.4 seconds inserted 10000000 Rows, about 746K ops/s