doris
doris copied to clipboard
Published
20 hours ago •
apache
apache
[refactor](query) Make query finish with context released safely
Proposed changes
We rely on a count number to decide if we can release a query context. However, count is not always reliable. If query is failed in preparation phase, the fragment number computed by FE may not be equal to its actual initialized fragments. So we here make query context a weak ptr which is release when all fragments finish instead of relying a reference count.
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
TeamCity be ut coverage result: Function Coverage: 35.66% (9015/25277) Line Coverage: 27.30% (74530/272987) Region Coverage: 26.53% (38572/145386) Branch Coverage: 23.39% (19676/84134) Coverage Report: http://coverage.selectdb-in.cc/coverage/caefcb003267e46cf00d47ac08e3065a9235fbee_caefcb003267e46cf00d47ac08e3065a9235fbee/report/index.html
TPC-H: Total hot run time: 41262 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit caefcb003267e46cf00d47ac08e3065a9235fbee, data reload: false
------ Round 1 ----------------------------------
q1 17600 4373 4231 4231
q2 2017 191 187 187
q3 10464 1212 1276 1212
q4 10186 897 783 783
q5 7470 2705 2705 2705
q6 223 134 133 133
q7 968 599 610 599
q8 9216 2146 2135 2135
q9 9153 6742 6686 6686
q10 9261 4001 3944 3944
q11 430 248 248 248
q12 421 220 221 220
q13 18661 3262 3086 3086
q14 264 237 220 220
q15 524 483 491 483
q16 519 409 402 402
q17 985 743 794 743
q18 8503 7871 7898 7871
q19 3528 1571 1534 1534
q20 652 316 338 316
q21 5155 3237 4134 3237
q22 360 287 302 287
Total cold run time: 116560 ms
Total hot run time: 41262 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4550 4366 4474 4366
q2 392 276 268 268
q3 3226 2810 2766 2766
q4 1870 1688 1666 1666
q5 5516 5492 5542 5492
q6 205 123 127 123
q7 2201 1865 1808 1808
q8 3207 3417 3415 3415
q9 8679 8676 8637 8637
q10 3907 3865 3898 3865
q11 600 505 511 505
q12 790 629 629 629
q13 17192 3199 3134 3134
q14 280 247 251 247
q15 530 473 478 473
q16 478 418 435 418
q17 1752 1509 1478 1478
q18 7888 7600 7593 7593
q19 1890 1579 1528 1528
q20 2005 1803 1792 1792
q21 9344 4654 4723 4654
q22 576 493 512 493
Total cold run time: 77078 ms
Total hot run time: 55350 ms
TPC-DS: Total hot run time: 169599 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit caefcb003267e46cf00d47ac08e3065a9235fbee, data reload: false
query1 913 384 378 378
query2 6457 2488 2302 2302
query3 6660 209 209 209
query4 19712 17334 17171 17171
query5 4150 428 409 409
query6 241 156 153 153
query7 4578 306 292 292
query8 245 188 192 188
query9 8433 2351 2334 2334
query10 465 295 265 265
query11 10865 9961 9905 9905
query12 136 92 87 87
query13 1642 368 385 368
query14 9165 7510 7436 7436
query15 215 172 167 167
query16 7829 268 267 267
query17 1696 546 531 531
query18 1998 268 262 262
query19 206 151 151 151
query20 97 93 91 91
query21 192 128 130 128
query22 4104 3881 3946 3881
query23 33905 33067 33211 33067
query24 6607 2849 2915 2849
query25 540 355 359 355
query26 712 157 157 157
query27 1941 321 313 313
query28 3710 2058 2058 2058
query29 863 620 605 605
query30 250 165 185 165
query31 958 735 772 735
query32 95 52 53 52
query33 492 267 262 262
query34 872 468 495 468
query35 693 617 597 597
query36 1067 909 928 909
query37 110 72 70 70
query38 2913 2799 2792 2792
query39 866 789 787 787
query40 199 132 127 127
query41 50 46 49 46
query42 104 96 100 96
query43 631 545 561 545
query44 1054 710 728 710
query45 190 164 167 164
query46 1070 715 714 714
query47 1845 1772 1745 1745
query48 358 300 292 292
query49 767 372 392 372
query50 772 385 387 385
query51 6963 6846 6888 6846
query52 109 89 88 88
query53 351 288 292 288
query54 535 429 430 429
query55 74 73 73 73
query56 270 249 241 241
query57 1121 1042 1018 1018
query58 235 202 244 202
query59 3490 3290 3316 3290
query60 275 261 247 247
query61 85 86 87 86
query62 578 456 451 451
query63 309 292 282 282
query64 8433 2224 1709 1709
query65 3123 3097 3114 3097
query66 782 326 329 326
query67 15213 14593 14765 14593
query68 4557 530 548 530
query69 439 273 268 268
query70 1156 1115 1038 1038
query71 394 275 268 268
query72 7591 5712 2715 2715
query73 731 335 321 321
query74 6115 5620 5687 5620
query75 3305 2635 2629 2629
query76 2174 924 1007 924
query77 380 274 259 259
query78 10297 9740 9753 9740
query79 2543 516 513 513
query80 1076 435 441 435
query81 527 247 241 241
query82 901 98 92 92
query83 256 179 171 171
query84 255 88 83 83
query85 1074 260 255 255
query86 447 308 308 308
query87 3262 3092 3072 3072
query88 4125 2380 2344 2344
query89 475 397 387 387
query90 2076 208 194 194
query91 136 169 99 99
query92 63 48 50 48
query93 1837 527 511 511
query94 1171 197 188 188
query95 414 319 317 317
query96 601 273 273 273
query97 3230 3018 3047 3018
query98 239 215 211 211
query99 1179 849 853 849
Total cold run time: 258605 ms
Total hot run time: 169599 ms
ClickBench: Total hot run time: 30.55 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit caefcb003267e46cf00d47ac08e3065a9235fbee, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.03 0.04
query3 0.23 0.05 0.05
query4 1.67 0.08 0.08
query5 0.50 0.50 0.52
query6 1.13 0.74 0.71
query7 0.02 0.01 0.01
query8 0.05 0.04 0.04
query9 0.54 0.49 0.48
query10 0.55 0.57 0.54
query11 0.16 0.11 0.11
query12 0.15 0.13 0.12
query13 0.59 0.59 0.60
query14 0.75 0.81 0.77
query15 0.82 0.80 0.81
query16 0.37 0.37 0.37
query17 1.01 1.02 1.02
query18 0.23 0.22 0.27
query19 1.89 1.74 1.75
query20 0.01 0.00 0.01
query21 15.50 0.69 0.66
query22 4.75 6.87 1.89
query23 18.32 1.43 1.27
query24 1.57 0.34 0.22
query25 0.14 0.08 0.07
query26 0.27 0.17 0.17
query27 0.07 0.08 0.07
query28 13.34 1.01 1.01
query29 13.34 3.33 3.34
query30 0.24 0.05 0.06
query31 2.88 0.39 0.38
query32 3.27 0.46 0.46
query33 2.90 2.97 2.88
query34 17.00 4.40 4.45
query35 4.49 4.43 4.73
query36 0.64 0.46 0.48
query37 0.18 0.15 0.16
query38 0.15 0.15 0.14
query39 0.04 0.04 0.03
query40 0.18 0.14 0.14
query41 0.09 0.04 0.04
query42 0.05 0.04 0.05
query43 0.04 0.04 0.04
Total cold run time: 110.24 s
Total hot run time: 30.55 s
TPC-H: Total hot run time: 41380 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c9ee12775683d2ce3e95e7e799521b9b5c9b96b4, data reload: false
------ Round 1 ----------------------------------
q1 17605 4395 4204 4204
q2 2023 185 193 185
q3 10457 1204 1208 1204
q4 10198 777 894 777
q5 7479 2827 2770 2770
q6 216 131 137 131
q7 972 613 614 613
q8 9263 2203 2125 2125
q9 9840 6816 6786 6786
q10 9457 3981 3852 3852
q11 448 247 242 242
q12 447 227 229 227
q13 17553 3268 3153 3153
q14 249 224 209 209
q15 518 479 465 465
q16 525 414 401 401
q17 980 699 735 699
q18 8534 7886 7940 7886
q19 3226 1583 1524 1524
q20 639 313 319 313
q21 5234 4089 3340 3340
q22 346 275 274 274
Total cold run time: 116209 ms
Total hot run time: 41380 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4545 4437 4381 4381
q2 378 292 287 287
q3 3199 2975 2771 2771
q4 1909 1568 1659 1568
q5 5498 5501 5517 5501
q6 219 127 126 126
q7 2165 1818 1863 1818
q8 3244 3431 3386 3386
q9 8612 8734 8631 8631
q10 3882 3784 3810 3784
q11 605 498 482 482
q12 797 647 637 637
q13 16710 3185 3195 3185
q14 298 276 268 268
q15 526 478 473 473
q16 479 414 424 414
q17 1781 1455 1467 1455
q18 7757 7620 7463 7463
q19 2251 1511 1585 1511
q20 2013 1787 1764 1764
q21 10350 4693 4717 4693
q22 553 491 502 491
Total cold run time: 77771 ms
Total hot run time: 55089 ms
TPC-DS: Total hot run time: 168485 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c9ee12775683d2ce3e95e7e799521b9b5c9b96b4, data reload: false
query1 919 393 378 378
query2 6443 2594 2383 2383
query3 6664 212 208 208
query4 20117 17361 17313 17313
query5 4157 421 412 412
query6 246 164 154 154
query7 4581 305 289 289
query8 242 189 181 181
query9 8417 2390 2345 2345
query10 451 271 258 258
query11 10507 10056 10061 10056
query12 131 96 87 87
query13 1646 371 371 371
query14 10078 6818 5942 5942
query15 201 172 174 172
query16 7717 277 257 257
query17 1369 514 531 514
query18 2206 270 264 264
query19 198 155 155 155
query20 93 84 83 83
query21 195 132 123 123
query22 4306 4099 3921 3921
query23 33522 33007 33168 33007
query24 9545 2828 2807 2807
query25 568 357 372 357
query26 702 158 159 158
query27 2200 313 322 313
query28 5852 2059 2049 2049
query29 860 611 591 591
query30 277 173 175 173
query31 969 750 752 750
query32 90 52 54 52
query33 661 284 306 284
query34 852 480 497 480
query35 704 597 608 597
query36 1084 920 926 920
query37 110 67 71 67
query38 2872 2788 2760 2760
query39 856 774 806 774
query40 199 122 122 122
query41 44 42 42 42
query42 103 97 97 97
query43 600 586 553 553
query44 1062 726 727 726
query45 181 163 163 163
query46 1067 736 723 723
query47 1865 1762 1775 1762
query48 368 299 309 299
query49 852 381 399 381
query50 769 392 375 375
query51 6870 6842 6799 6799
query52 110 86 91 86
query53 349 288 283 283
query54 822 424 419 419
query55 75 74 73 73
query56 257 238 259 238
query57 1128 1052 1036 1036
query58 229 208 213 208
query59 3517 3334 3243 3243
query60 281 254 249 249
query61 86 88 86 86
query62 602 459 446 446
query63 306 282 281 281
query64 8463 2277 1717 1717
query65 3156 3089 3080 3080
query66 773 350 321 321
query67 15106 14934 14869 14869
query68 4627 541 535 535
query69 509 267 271 267
query70 1133 1130 1138 1130
query71 392 262 258 258
query72 7921 2706 2567 2567
query73 714 332 327 327
query74 6069 5643 5577 5577
query75 3331 2628 2626 2626
query76 2933 972 947 947
query77 611 265 271 265
query78 10175 9716 9722 9716
query79 2284 524 522 522
query80 948 438 427 427
query81 544 238 242 238
query82 1060 100 96 96
query83 243 175 176 175
query84 249 84 86 84
query85 1474 271 261 261
query86 457 290 300 290
query87 3306 3096 3082 3082
query88 4156 2368 2374 2368
query89 500 396 379 379
query90 2018 197 191 191
query91 136 119 106 106
query92 62 50 50 50
query93 2427 518 577 518
query94 1282 186 182 182
query95 391 318 306 306
query96 589 265 268 265
query97 3197 3005 3005 3005
query98 260 219 215 215
query99 1138 854 828 828
Total cold run time: 266984 ms
Total hot run time: 168485 ms
TeamCity be ut coverage result: Function Coverage: 35.66% (9015/25277) Line Coverage: 27.31% (74527/272904) Region Coverage: 26.54% (38571/145328) Branch Coverage: 23.40% (19681/84094) Coverage Report: http://coverage.selectdb-in.cc/coverage/c9ee12775683d2ce3e95e7e799521b9b5c9b96b4_c9ee12775683d2ce3e95e7e799521b9b5c9b96b4/report/index.html
ClickBench: Total hot run time: 30.59 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c9ee12775683d2ce3e95e7e799521b9b5c9b96b4, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.04 0.04
query3 0.23 0.05 0.05
query4 1.68 0.06 0.06
query5 0.50 0.48 0.50
query6 1.15 0.73 0.72
query7 0.01 0.01 0.01
query8 0.05 0.04 0.04
query9 0.54 0.48 0.48
query10 0.54 0.56 0.55
query11 0.15 0.11 0.11
query12 0.15 0.11 0.12
query13 0.60 0.58 0.60
query14 0.77 0.77 0.76
query15 0.82 0.80 0.80
query16 0.37 0.35 0.35
query17 0.94 0.97 1.01
query18 0.23 0.21 0.27
query19 1.87 1.76 1.68
query20 0.01 0.01 0.01
query21 15.75 0.68 0.68
query22 4.47 7.15 2.04
query23 18.25 1.44 1.28
query24 1.71 0.30 0.22
query25 0.15 0.08 0.09
query26 0.27 0.17 0.17
query27 0.08 0.08 0.08
query28 13.25 1.02 0.99
query29 13.78 3.27 3.27
query30 0.24 0.06 0.06
query31 2.87 0.38 0.37
query32 3.29 0.46 0.46
query33 2.92 2.88 2.88
query34 17.21 4.41 4.43
query35 4.50 4.52 4.65
query36 0.65 0.46 0.46
query37 0.17 0.15 0.15
query38 0.16 0.14 0.14
query39 0.04 0.04 0.03
query40 0.16 0.14 0.14
query41 0.10 0.05 0.04
query42 0.06 0.05 0.05
query43 0.03 0.03 0.04
Total cold run time: 110.83 s
Total hot run time: 30.59 s
PR approved by at least one committer and no changes requested.