doris
doris copied to clipboard
apache
[opt](inverted index) performance optimization for need_read_data in compound #35346
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
TPC-H: Total hot run time: 49752 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4ebbddf45aa730797dfb56f0ce2d94f64d442eaf, data reload: false
------ Round 1 ----------------------------------
q1 17668 4358 4379 4358
q2 2081 153 143 143
q3 10451 1890 1946 1890
q4 10329 1219 1296 1219
q5 8480 3859 3909 3859
q6 232 127 128 127
q7 2036 1597 1625 1597
q8 9282 2704 2687 2687
q9 10541 10321 10270 10270
q10 8655 3515 3530 3515
q11 419 255 249 249
q12 465 307 302 302
q13 18347 3934 4055 3934
q14 353 324 332 324
q15 501 460 463 460
q16 670 589 573 573
q17 1121 969 970 969
q18 7316 6935 6951 6935
q19 1712 1584 1575 1575
q20 547 308 300 300
q21 4464 4094 4074 4074
q22 497 397 392 392
Total cold run time: 116167 ms
Total hot run time: 49752 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4305 4310 4311 4310
q2 323 230 222 222
q3 4160 4112 4104 4104
q4 2740 2732 2733 2732
q5 7180 7061 7076 7061
q6 239 117 120 117
q7 3231 2825 2868 2825
q8 4321 4434 4421 4421
q9 16879 16800 16641 16641
q10 4246 4261 4247 4247
q11 745 684 695 684
q12 1036 877 863 863
q13 6761 3736 3722 3722
q14 469 420 421 420
q15 501 447 449 447
q16 720 670 686 670
q17 3780 3978 3863 3863
q18 8757 8673 8695 8673
q19 1696 1698 1634 1634
q20 2381 2156 2108 2108
q21 8486 8376 8492 8376
q22 1032 974 959 959
Total cold run time: 83988 ms
Total hot run time: 79099 ms
TeamCity be ut coverage result: Function Coverage: 37.81% (8099/21421) Line Coverage: 29.47% (66147/224444) Region Coverage: 28.96% (34095/117738) Branch Coverage: 24.81% (17498/70524) Coverage Report: http://coverage.selectdb-in.cc/coverage/4ebbddf45aa730797dfb56f0ce2d94f64d442eaf_4ebbddf45aa730797dfb56f0ce2d94f64d442eaf/report/index.html
TPC-DS: Total hot run time: 203039 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ebbddf45aa730797dfb56f0ce2d94f64d442eaf, data reload: false
query1 932 382 374 374
query2 6599 2860 2603 2603
query3 6919 213 204 204
query4 20107 17923 17867 17867
query5 19722 6510 6503 6503
query6 349 215 232 215
query7 4613 299 303 299
query8 270 248 259 248
query9 3136 2681 2634 2634
query10 418 292 304 292
query11 11253 10643 10572 10572
query12 119 81 74 74
query13 5591 683 707 683
query14 18485 13527 13589 13527
query15 384 227 228 227
query16 6439 294 260 260
query17 1354 1563 887 887
query18 2252 407 414 407
query19 216 151 156 151
query20 80 80 80 80
query21 190 96 96 96
query22 5042 5087 4910 4910
query23 32664 31568 32254 31568
query24 6876 6571 6509 6509
query25 521 440 420 420
query26 510 168 160 160
query27 1780 292 297 292
query28 6132 2396 2347 2347
query29 2905 2816 2765 2765
query30 248 165 169 165
query31 916 759 741 741
query32 69 62 61 61
query33 404 264 244 244
query34 864 465 475 465
query35 1129 913 931 913
query36 1215 1066 1183 1066
query37 92 62 68 62
query38 3120 2929 2931 2929
query39 1372 1310 1312 1310
query40 209 99 95 95
query41 41 37 37 37
query42 91 80 83 80
query43 764 656 628 628
query44 1123 719 717 717
query45 244 229 230 229
query46 1243 967 977 967
query47 1950 1640 1787 1640
query48 1014 691 683 683
query49 626 372 363 363
query50 868 590 624 590
query51 4766 4687 4698 4687
query52 89 91 74 74
query53 448 311 318 311
query54 2661 2428 2445 2428
query55 89 82 83 82
query56 246 222 221 221
query57 1142 1097 1108 1097
query58 205 207 199 199
query59 4197 4115 3859 3859
query60 216 209 203 203
query61 97 97 105 97
query62 833 479 458 458
query63 479 328 338 328
query64 2583 1547 1508 1508
query65 3615 3547 3556 3547
query66 764 380 363 363
query67 15949 17075 15956 15956
query68 8487 674 645 645
query69 568 337 344 337
query70 1731 1741 1363 1363
query71 401 309 310 309
query72 6529 3496 3506 3496
query73 729 329 327 327
query74 6318 5721 5880 5721
query75 4664 3620 3678 3620
query76 4741 1178 1188 1178
query77 639 257 260 257
query78 12575 12037 11449 11449
query79 8744 668 650 650
query80 2245 413 402 402
query81 507 238 233 233
query82 1452 99 95 95
query83 170 137 133 133
query84 256 70 73 70
query85 1312 324 318 318
query86 365 294 290 290
query87 3184 2969 2988 2969
query88 5019 2345 2332 2332
query89 379 336 326 326
query90 1805 206 209 206
query91 175 140 140 140
query92 57 57 55 55
query93 5291 617 593 593
query94 893 207 218 207
query95 1105 1087 1057 1057
query96 640 341 329 329
query97 6444 6445 6299 6299
query98 190 172 172 172
query99 3002 884 985 884
Total cold run time: 312456 ms
Total hot run time: 203039 ms
ClickBench: Total hot run time: 31.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4ebbddf45aa730797dfb56f0ce2d94f64d442eaf, data reload: false
query1 0.02 0.03 0.02
query2 0.06 0.03 0.02
query3 0.25 0.04 0.05
query4 1.79 0.07 0.06
query5 0.54 0.53 0.52
query6 1.23 0.61 0.61
query7 0.02 0.01 0.00
query8 0.03 0.02 0.02
query9 0.52 0.48 0.46
query10 0.55 0.54 0.54
query11 0.12 0.08 0.08
query12 0.12 0.09 0.09
query13 0.61 0.62 0.61
query14 0.80 0.78 0.78
query15 0.77 0.76 0.77
query16 0.38 0.39 0.39
query17 0.95 0.99 0.99
query18 0.22 0.26 0.25
query19 1.88 1.80 1.87
query20 0.02 0.00 0.01
query21 15.48 0.55 0.57
query22 2.08 2.53 1.64
query23 17.19 0.99 0.99
query24 7.05 1.51 1.91
query25 0.38 0.11 0.05
query26 0.82 0.15 0.15
query27 0.04 0.04 0.04
query28 5.26 0.71 0.70
query29 12.61 2.42 2.37
query30 0.56 0.57 0.51
query31 2.80 0.39 0.38
query32 3.39 0.49 0.48
query33 3.05 3.06 3.06
query34 15.27 4.81 4.80
query35 4.84 4.80 4.84
query36 1.07 1.01 1.01
query37 0.06 0.05 0.04
query38 0.04 0.02 0.02
query39 0.02 0.01 0.01
query40 0.16 0.14 0.14
query41 0.07 0.01 0.02
query42 0.02 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 103.16 s
Total hot run time: 31.33 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit 4ebbddf45aa730797dfb56f0ce2d94f64d442eaf with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.2 seconds inserted 10000000 Rows, about 471K ops/s
TPC-H: Total hot run time: 49716 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa86a12ad0448567b5f84ad17d55062dd7b4473a, data reload: false
------ Round 1 ----------------------------------
q1 17669 4369 4350 4350
q2 2042 185 148 148
q3 10376 1880 1947 1880
q4 10365 1268 1321 1268
q5 8425 3885 3884 3884
q6 237 124 123 123
q7 2052 1605 1587 1587
q8 9309 2727 2695 2695
q9 10537 10288 10262 10262
q10 8634 3533 3492 3492
q11 436 251 250 250
q12 468 301 301 301
q13 18363 3941 4020 3941
q14 368 331 341 331
q15 527 462 460 460
q16 675 576 581 576
q17 1116 947 995 947
q18 7209 6926 6977 6926
q19 1716 1587 1560 1560
q20 535 319 293 293
q21 4378 4172 4044 4044
q22 491 402 398 398
Total cold run time: 115928 ms
Total hot run time: 49716 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4339 4300 4360 4300
q2 320 222 223 222
q3 4204 4131 4126 4126
q4 2753 2744 2744 2744
q5 7187 7082 7112 7082
q6 238 121 118 118
q7 3241 2830 2895 2830
q8 4364 4432 4459 4432
q9 16909 16729 16739 16729
q10 4253 4265 4238 4238
q11 771 689 663 663
q12 1024 860 838 838
q13 7256 3702 3744 3702
q14 452 415 432 415
q15 499 453 456 453
q16 745 678 672 672
q17 3863 3836 3839 3836
q18 8892 8834 8670 8670
q19 1722 1692 1633 1633
q20 2388 2142 2110 2110
q21 8481 8393 8467 8393
q22 1012 938 900 900
Total cold run time: 84913 ms
Total hot run time: 79106 ms
TeamCity be ut coverage result: Function Coverage: 37.81% (8099/21421) Line Coverage: 29.47% (66148/224444) Region Coverage: 28.96% (34095/117738) Branch Coverage: 24.81% (17496/70524) Coverage Report: http://coverage.selectdb-in.cc/coverage/fa86a12ad0448567b5f84ad17d55062dd7b4473a_fa86a12ad0448567b5f84ad17d55062dd7b4473a/report/index.html
TPC-DS: Total hot run time: 203799 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa86a12ad0448567b5f84ad17d55062dd7b4473a, data reload: false
query1 929 389 378 378
query2 6554 2598 2481 2481
query3 6925 205 201 201
query4 20407 18114 18074 18074
query5 19728 6519 6547 6519
query6 277 215 222 215
query7 4144 303 311 303
query8 254 262 245 245
query9 3114 2679 2606 2606
query10 425 292 297 292
query11 11332 10696 10720 10696
query12 125 78 75 75
query13 5587 724 681 681
query14 17345 13355 14001 13355
query15 375 219 237 219
query16 6472 288 265 265
query17 1711 1460 874 874
query18 2320 413 405 405
query19 211 150 147 147
query20 71 82 78 78
query21 201 96 98 96
query22 5322 5143 5226 5143
query23 32494 31850 31766 31766
query24 7033 6585 6491 6491
query25 535 433 412 412
query26 628 164 157 157
query27 2045 298 294 294
query28 6121 2362 2314 2314
query29 2883 2768 2840 2768
query30 247 163 170 163
query31 905 739 735 735
query32 70 62 60 60
query33 406 255 241 241
query34 842 478 477 477
query35 1132 951 932 932
query36 1213 1303 1329 1303
query37 92 62 61 61
query38 3062 2923 2911 2911
query39 1390 1316 1339 1316
query40 220 91 97 91
query41 40 38 37 37
query42 84 81 84 81
query43 788 628 627 627
query44 1125 721 714 714
query45 242 232 230 230
query46 1230 960 956 956
query47 2062 1758 1701 1701
query48 1010 726 718 718
query49 625 381 363 363
query50 856 576 595 576
query51 4849 4623 4605 4605
query52 100 84 86 84
query53 444 320 327 320
query54 2675 2449 2444 2444
query55 79 80 78 78
query56 229 208 214 208
query57 1354 1163 1100 1100
query58 217 205 190 190
query59 4228 4066 3963 3963
query60 207 217 215 215
query61 94 93 94 93
query62 829 421 451 421
query63 487 342 336 336
query64 2498 1516 1482 1482
query65 3639 4081 3539 3539
query66 808 386 371 371
query67 17655 16210 15347 15347
query68 8552 656 689 656
query69 579 360 350 350
query70 1551 1339 1324 1324
query71 405 298 334 298
query72 6484 3467 3649 3467
query73 747 322 329 322
query74 6353 5862 5875 5862
query75 4772 3719 3694 3694
query76 4733 1170 1210 1170
query77 649 252 275 252
query78 12518 12555 12081 12081
query79 11881 671 619 619
query80 872 401 397 397
query81 504 235 239 235
query82 643 99 102 99
query83 173 134 134 134
query84 258 70 72 70
query85 867 321 328 321
query86 339 292 289 289
query87 3272 3044 3032 3032
query88 4926 2342 2348 2342
query89 363 282 304 282
query90 1892 206 211 206
query91 172 139 141 139
query92 58 54 54 54
query93 4243 582 557 557
query94 746 211 203 203
query95 1117 1062 1062 1062
query96 639 330 332 330
query97 6427 6343 6343 6343
query98 193 180 173 173
query99 2895 864 797 797
Total cold run time: 313424 ms
Total hot run time: 203799 ms
ClickBench: Total hot run time: 30.58 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fa86a12ad0448567b5f84ad17d55062dd7b4473a, data reload: false
query1 0.03 0.03 0.02
query2 0.07 0.02 0.02
query3 0.25 0.05 0.05
query4 1.78 0.08 0.08
query5 0.54 0.52 0.52
query6 1.24 0.62 0.63
query7 0.02 0.01 0.01
query8 0.03 0.02 0.02
query9 0.53 0.49 0.48
query10 0.53 0.54 0.53
query11 0.13 0.09 0.09
query12 0.11 0.09 0.09
query13 0.62 0.62 0.62
query14 0.79 0.79 0.78
query15 0.78 0.77 0.76
query16 0.36 0.36 0.36
query17 1.01 1.01 0.99
query18 0.24 0.23 0.26
query19 1.88 1.88 1.82
query20 0.02 0.01 0.01
query21 15.84 0.57 0.57
query22 2.38 2.57 1.61
query23 16.86 1.02 1.07
query24 7.12 0.83 0.92
query25 0.38 0.12 0.05
query26 0.68 0.15 0.15
query27 0.05 0.04 0.03
query28 6.27 0.73 0.74
query29 12.72 2.13 2.17
query30 0.60 0.55 0.52
query31 2.82 0.38 0.37
query32 3.38 0.51 0.49
query33 3.07 3.05 3.11
query34 15.27 4.80 4.80
query35 4.85 4.87 4.87
query36 1.05 1.03 1.01
query37 0.06 0.05 0.04
query38 0.03 0.02 0.02
query39 0.02 0.02 0.02
query40 0.16 0.14 0.14
query41 0.06 0.01 0.01
query42 0.02 0.01 0.02
query43 0.02 0.02 0.01
Total cold run time: 104.67 s
Total hot run time: 30.58 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit fa86a12ad0448567b5f84ad17d55062dd7b4473a with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.3 seconds inserted 10000000 Rows, about 469K ops/s