doris
doris copied to clipboard
Published
20 hours ago •
apache
apache
[fix](move-memtable) clear load streams before shutdown SegmentFileWriterThreadPool
Proposed changes
In ~LoadStreamMgr(), clear load streams before shutdown SegmentFileWriterThreadPool.
To fix the following check failure.
F20240428 03:36:59.966398 10071 threadpool.cpp:253] Check failed: 1 == _tokens.size() (1 vs. 97) Threadpool SegmentFileWriterThreadPool destroyed with 97 allocated tokens
*** Check failure stack trace: ***
@ 0x55a0d476b7a6 google::LogMessage::SendToLog()
@ 0x55a0d47681f0 google::LogMessage::Flush()
@ 0x55a0d476bfe9 google::LogMessageFatal::~LogMessageFatal()
@ 0x55a0cad50d19 doris::ThreadPool::~ThreadPool()
@ 0x55a0cac64f8b doris::LoadStreamMgr::~LoadStreamMgr()
@ 0x55a0cac36d02 doris::PInternalServiceImpl::~PInternalServiceImpl()
@ 0x55a0cac372ee doris::PInternalServiceImpl::~PInternalServiceImpl()
@ 0x55a0d63604f5 brpc::Server::ClearServices()
@ 0x55a0cac2bab1 doris::BRpcService::join()
@ 0x55a0cac2b9cf doris::BRpcService::~BRpcService()
@ 0x55a0c9e3ef7b main
@ 0x7f24519bd083 __libc_start_main
@ 0x55a0c9e3d02a _start
@ (nil) (unknown)
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
PR approved by at least one committer and no changes requested.
TPC-H: Total hot run time: 40533 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 081476715a10c658254339a0b89fd6d8742c5fe6, data reload: false
------ Round 1 ----------------------------------
q1 17912 4405 4254 4254
q2 2030 196 190 190
q3 10458 1209 1283 1209
q4 10161 796 761 761
q5 7456 2642 2625 2625
q6 214 131 133 131
q7 935 605 593 593
q8 9225 2047 2064 2047
q9 8853 6444 6455 6444
q10 8959 3675 3711 3675
q11 461 246 235 235
q12 478 220 228 220
q13 17771 2993 2984 2984
q14 277 228 229 228
q15 517 477 470 470
q16 535 390 377 377
q17 977 696 712 696
q18 8093 7500 7418 7418
q19 4899 1545 1519 1519
q20 668 312 313 312
q21 4991 3865 4038 3865
q22 350 280 283 280
Total cold run time: 116220 ms
Total hot run time: 40533 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4299 4208 4210 4208
q2 371 271 272 271
q3 2962 2743 2791 2743
q4 1873 1544 1619 1544
q5 5246 5251 5254 5251
q6 210 126 128 126
q7 2096 1715 1749 1715
q8 3173 3301 3337 3301
q9 8419 8343 8296 8296
q10 3888 3641 3640 3640
q11 609 486 493 486
q12 761 620 624 620
q13 16515 3034 2975 2975
q14 298 254 257 254
q15 527 468 478 468
q16 467 420 415 415
q17 1770 1507 1453 1453
q18 7657 7539 7497 7497
q19 1669 1557 1607 1557
q20 2004 1775 1809 1775
q21 4878 4829 4734 4734
q22 554 470 497 470
Total cold run time: 70246 ms
Total hot run time: 53799 ms
TeamCity be ut coverage result: Function Coverage: 35.66% (9013/25277) Line Coverage: 27.30% (74510/272956) Region Coverage: 26.52% (38554/145355) Branch Coverage: 23.38% (19665/84116) Coverage Report: http://coverage.selectdb-in.cc/coverage/081476715a10c658254339a0b89fd6d8742c5fe6_081476715a10c658254339a0b89fd6d8742c5fe6/report/index.html
TPC-DS: Total hot run time: 170907 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 081476715a10c658254339a0b89fd6d8742c5fe6, data reload: false
query1 915 383 371 371
query2 6456 2296 2248 2248
query3 6659 214 204 204
query4 19256 17223 17246 17223
query5 4142 412 410 410
query6 255 160 157 157
query7 4596 297 293 293
query8 233 183 190 183
query9 8514 2390 2380 2380
query10 458 291 264 264
query11 10413 10236 9974 9974
query12 132 88 90 88
query13 1641 354 353 353
query14 9256 7531 6212 6212
query15 281 172 187 172
query16 8156 263 251 251
query17 1863 538 541 538
query18 2073 265 269 265
query19 203 149 154 149
query20 95 89 94 89
query21 200 121 123 121
query22 4123 4062 4032 4032
query23 33720 33006 33027 33006
query24 11841 2822 2755 2755
query25 619 346 372 346
query26 1589 154 158 154
query27 2987 319 343 319
query28 7292 2046 2047 2046
query29 970 599 604 599
query30 311 168 172 168
query31 994 756 744 744
query32 92 51 51 51
query33 754 263 258 258
query34 1006 487 482 482
query35 733 592 579 579
query36 1075 917 889 889
query37 147 95 72 72
query38 2890 2751 2743 2743
query39 827 778 776 776
query40 278 128 124 124
query41 46 45 47 45
query42 101 96 96 96
query43 551 531 521 521
query44 1207 719 737 719
query45 183 159 167 159
query46 1069 709 713 709
query47 1850 1803 1752 1752
query48 367 289 288 288
query49 1135 372 380 372
query50 768 388 407 388
query51 6882 7099 6793 6793
query52 99 89 89 89
query53 348 301 289 289
query54 951 431 433 431
query55 75 72 71 71
query56 260 254 284 254
query57 1136 1053 1024 1024
query58 237 204 204 204
query59 3345 3280 2963 2963
query60 274 249 240 240
query61 87 85 102 85
query62 644 451 449 449
query63 307 285 291 285
query64 9847 2217 1722 1722
query65 3164 3095 3126 3095
query66 1355 332 319 319
query67 15264 14811 14850 14811
query68 4595 541 544 541
query69 443 265 264 264
query70 1137 1113 1051 1051
query71 416 265 263 263
query72 7721 5823 5601 5601
query73 725 367 316 316
query74 5927 5621 5576 5576
query75 3360 2649 2573 2573
query76 2625 961 969 961
query77 400 266 272 266
query78 10234 9759 9891 9759
query79 2633 509 503 503
query80 1068 445 435 435
query81 524 241 239 239
query82 767 94 95 94
query83 240 165 165 165
query84 237 84 86 84
query85 1543 260 263 260
query86 497 271 319 271
query87 3399 3099 3107 3099
query88 4320 2321 2339 2321
query89 493 387 388 387
query90 1976 193 191 191
query91 194 94 95 94
query92 65 49 50 49
query93 1758 508 493 493
query94 1197 193 188 188
query95 406 313 306 306
query96 582 260 267 260
query97 3158 2984 3004 2984
query98 230 228 226 226
query99 1098 870 826 826
Total cold run time: 273565 ms
Total hot run time: 170907 ms
ClickBench: Total hot run time: 30.78 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 081476715a10c658254339a0b89fd6d8742c5fe6, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.07
query4 1.66 0.08 0.09
query5 0.50 0.50 0.51
query6 1.12 0.72 0.72
query7 0.02 0.02 0.02
query8 0.05 0.03 0.04
query9 0.54 0.48 0.48
query10 0.54 0.53 0.53
query11 0.16 0.11 0.11
query12 0.15 0.12 0.11
query13 0.59 0.59 0.60
query14 0.78 0.79 0.78
query15 0.83 0.81 0.81
query16 0.35 0.37 0.36
query17 1.02 0.98 0.95
query18 0.21 0.27 0.25
query19 1.82 1.68 1.67
query20 0.02 0.01 0.00
query21 15.74 0.66 0.66
query22 4.05 7.60 2.07
query23 18.27 1.39 1.31
query24 2.05 0.23 0.21
query25 0.15 0.08 0.07
query26 0.25 0.17 0.17
query27 0.08 0.09 0.08
query28 13.26 1.10 1.09
query29 12.56 3.30 3.32
query30 0.24 0.06 0.05
query31 2.86 0.39 0.38
query32 3.27 0.48 0.48
query33 2.91 2.88 2.84
query34 17.09 4.42 4.43
query35 4.44 4.57 4.50
query36 0.65 0.46 0.46
query37 0.18 0.15 0.17
query38 0.16 0.14 0.14
query39 0.05 0.03 0.03
query40 0.16 0.14 0.15
query41 0.09 0.04 0.05
query42 0.06 0.04 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.32 s
Total hot run time: 30.78 s