doris
doris copied to clipboard
Published
20 hours ago •
apache
apache
[fix](index) should not use light index change for bloom filter index
cherry-pick https://github.com/apache/doris/pull/35214 to branch2.0
Problem:
- add or drop Bitmap Index;
- add or drop Bloom Filter Index; Both do not work correctly for history data.
related pr: https://github.com/apache/doris/pull/28135 https://github.com/apache/doris/pull/32839 https://github.com/apache/doris/pull/10136
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
TPC-H: Total hot run time: 49862 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 764f13438b4ca87edfdf593a47767c7d1d3275c9, data reload: false
------ Round 1 ----------------------------------
q1 17720 4415 4361 4361
q2 2065 157 151 151
q3 10538 1913 1957 1913
q4 10336 1258 1326 1258
q5 8608 3979 3913 3913
q6 234 121 124 121
q7 2041 1571 1574 1571
q8 9583 2750 2712 2712
q9 14081 10405 10456 10405
q10 8638 3554 3532 3532
q11 418 250 253 250
q12 468 299 293 293
q13 18369 3944 4043 3944
q14 349 325 334 325
q15 509 457 461 457
q16 668 563 569 563
q17 1157 984 951 951
q18 7341 6791 6870 6791
q19 1733 1559 1594 1559
q20 548 304 315 304
q21 4456 4133 4111 4111
q22 490 380 377 377
Total cold run time: 120350 ms
Total hot run time: 49862 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4298 4253 4268 4253
q2 313 217 222 217
q3 4137 4181 4139 4139
q4 2758 2743 2739 2739
q5 7216 7135 7126 7126
q6 235 118 119 118
q7 3221 2807 2788 2788
q8 4348 4441 4459 4441
q9 17369 17003 16882 16882
q10 4236 4253 4270 4253
q11 745 644 682 644
q12 1017 849 851 849
q13 6787 3763 3741 3741
q14 456 429 427 427
q15 494 463 448 448
q16 738 679 689 679
q17 3720 3863 3825 3825
q18 8929 8832 8758 8758
q19 1742 1692 1679 1679
q20 2417 2112 2099 2099
q21 8457 8281 8443 8281
q22 1004 970 959 959
Total cold run time: 84637 ms
Total hot run time: 79345 ms
TeamCity be ut coverage result: Function Coverage: 37.82% (8082/21372) Line Coverage: 29.47% (65992/223941) Region Coverage: 28.94% (33982/117425) Branch Coverage: 24.80% (17442/70338) Coverage Report: http://coverage.selectdb-in.cc/coverage/764f13438b4ca87edfdf593a47767c7d1d3275c9_764f13438b4ca87edfdf593a47767c7d1d3275c9/report/index.html
TPC-DS: Total hot run time: 203796 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 764f13438b4ca87edfdf593a47767c7d1d3275c9, data reload: false
query1 932 382 375 375
query2 6690 2704 2626 2626
query3 6923 212 210 210
query4 20429 17890 18015 17890
query5 19719 6507 6519 6507
query6 289 222 238 222
query7 4161 297 299 297
query8 272 229 247 229
query9 3126 2700 2599 2599
query10 425 293 294 293
query11 11422 10712 10777 10712
query12 119 78 72 72
query13 5597 689 694 689
query14 17813 13611 13362 13362
query15 360 215 221 215
query16 6455 273 259 259
query17 1754 1467 891 891
query18 2292 400 406 400
query19 211 150 153 150
query20 84 79 80 79
query21 192 105 93 93
query22 5233 5094 4973 4973
query23 32452 31908 31793 31793
query24 6913 6560 6515 6515
query25 507 427 431 427
query26 530 160 156 156
query27 1892 295 291 291
query28 6168 2342 2310 2310
query29 2883 2755 2856 2755
query30 243 165 162 162
query31 905 727 722 722
query32 65 60 64 60
query33 403 246 249 246
query34 875 470 483 470
query35 1123 881 944 881
query36 1334 1097 1182 1097
query37 93 58 64 58
query38 3113 2923 2906 2906
query39 1385 1310 1325 1310
query40 207 95 93 93
query41 38 40 36 36
query42 78 86 77 77
query43 742 760 649 649
query44 1164 710 718 710
query45 240 230 230 230
query46 1228 951 997 951
query47 1864 1702 1670 1670
query48 1007 707 709 707
query49 612 366 376 366
query50 858 608 598 598
query51 4772 4696 4637 4637
query52 88 83 75 75
query53 445 317 313 313
query54 2663 2464 2463 2463
query55 85 88 77 77
query56 212 223 217 217
query57 1149 1062 1099 1062
query58 222 203 199 199
query59 4011 4120 3942 3942
query60 205 194 211 194
query61 90 87 85 85
query62 886 469 507 469
query63 477 342 343 342
query64 2531 1521 1506 1506
query65 3593 3551 3563 3551
query66 807 388 365 365
query67 15866 15891 17196 15891
query68 8861 670 658 658
query69 583 353 364 353
query70 1588 1344 1449 1344
query71 413 306 305 305
query72 6459 3459 3474 3459
query73 741 321 328 321
query74 6355 5818 5849 5818
query75 5275 3720 3690 3690
query76 5297 1160 1202 1160
query77 898 256 250 250
query78 12726 13144 11972 11972
query79 9244 635 648 635
query80 808 391 378 378
query81 484 233 231 231
query82 370 95 100 95
query83 167 130 134 130
query84 259 70 69 69
query85 692 299 289 289
query86 342 308 315 308
query87 3261 2953 3053 2953
query88 4731 2337 2350 2337
query89 425 276 279 276
query90 1883 210 209 209
query91 167 134 134 134
query92 59 50 51 50
query93 4162 581 562 562
query94 698 202 206 202
query95 1096 1049 1053 1049
query96 648 333 322 322
query97 6608 6387 6433 6387
query98 190 174 177 174
query99 2757 915 899 899
Total cold run time: 309769 ms
Total hot run time: 203796 ms
ClickBench: Total hot run time: 30.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 764f13438b4ca87edfdf593a47767c7d1d3275c9, data reload: false
query1 0.02 0.02 0.02
query2 0.08 0.02 0.02
query3 0.26 0.05 0.05
query4 1.77 0.06 0.06
query5 0.53 0.53 0.52
query6 1.25 0.62 0.62
query7 0.01 0.01 0.01
query8 0.03 0.03 0.02
query9 0.52 0.48 0.48
query10 0.54 0.54 0.53
query11 0.12 0.08 0.09
query12 0.12 0.09 0.09
query13 0.62 0.62 0.60
query14 0.79 0.77 0.79
query15 0.79 0.76 0.76
query16 0.36 0.37 0.38
query17 1.02 0.96 1.02
query18 0.21 0.27 0.24
query19 1.87 1.82 1.86
query20 0.01 0.01 0.01
query21 15.47 0.56 0.57
query22 2.14 2.04 1.54
query23 16.89 0.87 0.92
query24 7.15 1.32 0.98
query25 0.40 0.11 0.04
query26 0.71 0.15 0.16
query27 0.05 0.03 0.03
query28 5.70 0.78 0.72
query29 12.63 2.42 2.34
query30 0.54 0.49 0.53
query31 2.83 0.38 0.38
query32 3.37 0.50 0.50
query33 3.05 3.05 3.09
query34 15.24 4.81 4.81
query35 4.87 4.85 4.84
query36 1.06 1.02 1.01
query37 0.06 0.05 0.05
query38 0.04 0.02 0.03
query39 0.02 0.01 0.02
query40 0.16 0.15 0.14
query41 0.07 0.01 0.01
query42 0.02 0.02 0.02
query43 0.02 0.02 0.02
Total cold run time: 103.41 s
Total hot run time: 30.61 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit 764f13438b4ca87edfdf593a47767c7d1d3275c9 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 21.6 seconds inserted 10000000 Rows, about 462K ops/s