doris
doris copied to clipboard
Published
20 hours ago •
apache
apache
[branch-2.0][improvement](mysql catalog) disable mysql AbandonedConnectionCleanup Thread
pick (#36655)
When using mysql catalog, mysql jdbc driver will generate an AbandonedConnectionCleanupThread for each database connection. This is a virtual reference, which will accumulate over time as database connections are constantly created, eventually causing OOM. Therefore, in our usage scenario, we need to turn off this thread because our database connection recycling depends on the connection pool. But please note that this switch is only for MySQL JDBC Driver versions greater than 8.0.22
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
TPC-H: Total hot run time: 50039 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f2a70ac6c9bb11505fb787e11b3d7d3352728f7, data reload: false
------ Round 1 ----------------------------------
q1 17897 4353 4340 4340
q2 2040 156 151 151
q3 10421 1881 1944 1881
q4 10312 1291 1305 1291
q5 8458 3924 3864 3864
q6 234 126 126 126
q7 2033 1590 1599 1590
q8 9288 2717 2689 2689
q9 10961 10599 10489 10489
q10 8656 3523 3513 3513
q11 423 247 255 247
q12 471 310 316 310
q13 18351 3983 4028 3983
q14 351 317 326 317
q15 503 463 461 461
q16 680 569 571 569
q17 1132 949 974 949
q18 7266 6875 6856 6856
q19 1751 1661 1600 1600
q20 560 305 308 305
q21 4414 4092 4074 4074
q22 528 434 451 434
Total cold run time: 116730 ms
Total hot run time: 50039 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4349 4269 4286 4269
q2 323 233 232 232
q3 4166 4126 4138 4126
q4 2771 2723 2738 2723
q5 7227 7099 7041 7041
q6 238 121 120 120
q7 3286 2879 2836 2836
q8 4354 4485 4471 4471
q9 17335 16744 17001 16744
q10 4230 4293 4261 4261
q11 734 705 674 674
q12 1053 859 856 856
q13 6696 3703 3732 3703
q14 443 428 418 418
q15 507 458 464 458
q16 736 693 691 691
q17 3840 3900 3835 3835
q18 8647 8704 8756 8704
q19 1713 1700 1700 1700
q20 2375 2129 2063 2063
q21 8408 8421 8495 8421
q22 1111 1067 1057 1057
Total cold run time: 84542 ms
Total hot run time: 79403 ms
TPC-DS: Total hot run time: 202971 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f2a70ac6c9bb11505fb787e11b3d7d3352728f7, data reload: false
query1 952 431 383 383
query2 7111 2566 2625 2566
query3 6919 207 203 203
query4 20388 17869 17813 17813
query5 19726 6514 6563 6514
query6 320 214 230 214
query7 4298 307 326 307
query8 429 418 420 418
query9 3201 2708 2631 2631
query10 420 315 281 281
query11 11257 10685 10788 10685
query12 129 79 75 75
query13 5608 703 685 685
query14 18693 13693 13008 13008
query15 365 248 251 248
query16 6443 287 265 265
query17 1486 1503 891 891
query18 2253 420 413 413
query19 210 151 151 151
query20 83 80 82 80
query21 197 101 107 101
query22 5321 5152 5048 5048
query23 32546 32131 31884 31884
query24 6813 6624 6503 6503
query25 538 462 425 425
query26 526 176 162 162
query27 1831 294 309 294
query28 6083 2383 2338 2338
query29 2848 2719 2791 2719
query30 244 164 169 164
query31 900 726 745 726
query32 81 64 63 63
query33 405 253 250 250
query34 873 468 481 468
query35 1120 924 949 924
query36 1326 1145 1124 1124
query37 94 60 61 60
query38 3129 2963 2919 2919
query39 1371 1348 1329 1329
query40 204 96 100 96
query41 46 45 42 42
query42 90 86 91 86
query43 754 698 695 695
query44 1164 720 723 720
query45 249 243 241 241
query46 1248 994 989 989
query47 1978 1698 1817 1698
query48 1026 729 731 729
query49 626 386 378 378
query50 872 635 581 581
query51 4750 4726 4668 4668
query52 90 98 84 84
query53 455 335 326 326
query54 2655 2461 2458 2458
query55 87 85 86 85
query56 243 220 224 220
query57 1235 1152 1075 1075
query58 222 220 216 216
query59 4543 3762 3970 3762
query60 211 207 224 207
query61 101 99 101 99
query62 822 491 452 452
query63 496 343 358 343
query64 2448 1555 1449 1449
query65 3648 3573 3625 3573
query66 813 396 385 385
query67 15445 15977 15223 15223
query68 10576 628 649 628
query69 584 336 354 336
query70 2225 1199 1352 1199
query71 424 307 324 307
query72 6516 3541 3551 3541
query73 737 327 312 312
query74 6341 5869 5863 5863
query75 5522 3684 3706 3684
query76 6851 1134 1211 1134
query77 1185 257 254 254
query78 12922 11739 14819 11739
query79 7003 634 632 632
query80 810 419 415 415
query81 494 237 237 237
query82 492 97 94 94
query83 171 137 132 132
query84 263 72 72 72
query85 876 323 325 323
query86 336 298 306 298
query87 3201 2990 3012 2990
query88 3866 2278 2293 2278
query89 343 286 304 286
query90 1976 206 213 206
query91 177 141 139 139
query92 60 56 54 54
query93 3067 546 552 546
query94 711 209 205 205
query95 1113 1087 1077 1077
query96 640 328 318 318
query97 6514 6501 6444 6444
query98 195 170 178 170
query99 2861 847 846 846
Total cold run time: 312110 ms
Total hot run time: 202971 ms
ClickBench: Total hot run time: 31.42 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f2a70ac6c9bb11505fb787e11b3d7d3352728f7, data reload: false
query1 0.02 0.02 0.02
query2 0.07 0.02 0.02
query3 0.25 0.04 0.06
query4 1.75 0.10 0.10
query5 0.53 0.52 0.52
query6 1.28 0.62 0.61
query7 0.02 0.01 0.01
query8 0.03 0.02 0.02
query9 0.52 0.48 0.48
query10 0.53 0.54 0.53
query11 0.12 0.08 0.09
query12 0.12 0.09 0.09
query13 0.63 0.62 0.62
query14 0.78 0.78 0.79
query15 0.77 0.75 0.78
query16 0.36 0.35 0.35
query17 1.00 1.01 1.02
query18 0.21 0.25 0.25
query19 1.92 1.88 1.77
query20 0.02 0.01 0.01
query21 15.46 0.57 0.54
query22 2.04 2.49 1.57
query23 16.94 0.86 1.12
query24 6.07 1.92 2.00
query25 0.35 0.10 0.05
query26 0.86 0.16 0.18
query27 0.04 0.04 0.04
query28 5.42 0.78 0.73
query29 12.64 2.13 2.28
query30 0.55 0.52 0.51
query31 2.80 0.39 0.37
query32 3.37 0.50 0.50
query33 3.07 3.04 3.06
query34 15.26 4.86 4.80
query35 4.82 4.88 4.88
query36 1.06 1.01 1.01
query37 0.06 0.05 0.05
query38 0.04 0.02 0.02
query39 0.02 0.02 0.01
query40 0.17 0.14 0.14
query41 0.06 0.02 0.01
query42 0.02 0.01 0.02
query43 0.03 0.02 0.01
Total cold run time: 102.08 s
Total hot run time: 31.42 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit 3f2a70ac6c9bb11505fb787e11b3d7d3352728f7 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.4 seconds inserted 10000000 Rows, about 467K ops/s
TPC-H: Total hot run time: 50550 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4321fae75262e0099fd96d3ff53c3209ff3d11d7, data reload: false
------ Round 1 ----------------------------------
q1 18124 4442 4398 4398
q2 2073 155 147 147
q3 10412 1918 1944 1918
q4 10601 1241 1365 1241
q5 9245 3910 3925 3910
q6 228 149 128 128
q7 2060 1578 1600 1578
q8 9520 2741 2721 2721
q9 11037 10856 10716 10716
q10 8933 3702 3702 3702
q11 434 263 264 263
q12 480 304 310 304
q13 19772 4006 4042 4006
q14 378 338 328 328
q15 524 480 469 469
q16 818 595 648 595
q17 1524 976 946 946
q18 7316 6898 6754 6754
q19 1768 1608 1571 1571
q20 545 307 316 307
q21 4371 4115 4146 4115
q22 544 446 433 433
Total cold run time: 120707 ms
Total hot run time: 50550 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4263 4267 4281 4267
q2 321 225 218 218
q3 4152 4166 4110 4110
q4 2742 2729 2734 2729
q5 7212 7132 7108 7108
q6 236 121 118 118
q7 3209 2755 2826 2755
q8 4315 4430 4425 4425
q9 17366 17216 17299 17216
q10 4366 4310 4226 4226
q11 762 675 678 675
q12 1048 850 862 850
q13 6998 3720 3738 3720
q14 447 424 413 413
q15 509 465 463 463
q16 742 683 675 675
q17 3874 3784 3795 3784
q18 8750 8607 8735 8607
q19 1704 1701 1639 1639
q20 2372 2082 2090 2082
q21 8468 8494 8424 8424
q22 1073 981 951 951
Total cold run time: 84929 ms
Total hot run time: 79455 ms
TPC-DS: Total hot run time: 203407 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4321fae75262e0099fd96d3ff53c3209ff3d11d7, data reload: false
query1 936 384 427 384
query2 6551 2696 2603 2603
query3 6931 204 207 204
query4 20778 17984 17941 17941
query5 19746 6498 6535 6498
query6 320 224 231 224
query7 4389 294 295 294
query8 440 421 407 407
query9 3113 2678 2605 2605
query10 424 311 312 311
query11 11398 10623 10644 10623
query12 119 72 74 72
query13 5591 691 664 664
query14 17997 13179 13482 13179
query15 357 245 238 238
query16 6447 293 261 261
query17 1412 1538 858 858
query18 2290 408 417 408
query19 212 144 150 144
query20 78 78 79 78
query21 193 95 92 92
query22 5149 5068 5026 5026
query23 32492 31907 31863 31863
query24 6733 6511 6502 6502
query25 531 440 417 417
query26 519 161 157 157
query27 1839 287 292 287
query28 6090 2370 2308 2308
query29 2813 2693 2733 2693
query30 245 165 167 165
query31 896 729 730 729
query32 70 63 62 62
query33 403 244 260 244
query34 859 456 476 456
query35 1118 919 891 891
query36 1352 1126 1220 1126
query37 97 59 59 59
query38 3114 2900 2931 2900
query39 1385 1345 1336 1336
query40 210 98 97 97
query41 46 44 44 44
query42 80 80 77 77
query43 691 702 787 702
query44 1129 709 711 709
query45 254 234 235 234
query46 1219 967 963 963
query47 2028 1719 1906 1719
query48 1024 692 693 692
query49 623 358 378 358
query50 867 578 565 565
query51 4882 4702 4623 4623
query52 88 90 85 85
query53 446 313 323 313
query54 2667 2436 2426 2426
query55 90 79 83 79
query56 231 221 204 204
query57 1296 1095 1134 1095
query58 218 202 207 202
query59 4047 4046 3816 3816
query60 216 189 214 189
query61 100 99 95 95
query62 850 485 466 466
query63 476 334 337 334
query64 2507 1554 1508 1508
query65 3625 3560 3535 3535
query66 763 383 369 369
query67 15568 15826 15807 15807
query68 9335 626 661 626
query69 567 342 358 342
query70 1771 1309 1391 1309
query71 428 306 312 306
query72 6659 3546 3486 3486
query73 739 327 329 327
query74 6358 5865 5836 5836
query75 5416 3721 3726 3721
query76 5894 1129 1195 1129
query77 1014 258 268 258
query78 12642 11764 11685 11685
query79 6785 632 637 632
query80 956 406 408 406
query81 486 234 235 234
query82 1533 102 98 98
query83 178 142 135 135
query84 261 74 73 73
query85 880 330 328 328
query86 332 302 312 302
query87 3233 3008 2999 2999
query88 4592 2341 2334 2334
query89 373 328 308 308
query90 1974 214 218 214
query91 172 144 140 140
query92 58 53 54 53
query93 4068 573 583 573
query94 715 210 216 210
query95 1096 1067 1055 1055
query96 645 327 329 327
query97 6499 6343 6342 6342
query98 197 173 167 167
query99 2956 837 980 837
Total cold run time: 310480 ms
Total hot run time: 203407 ms
ClickBench: Total hot run time: 31.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4321fae75262e0099fd96d3ff53c3209ff3d11d7, data reload: false
query1 0.02 0.02 0.02
query2 0.07 0.02 0.02
query3 0.25 0.05 0.04
query4 1.81 0.06 0.05
query5 0.53 0.52 0.52
query6 1.27 0.62 0.62
query7 0.01 0.01 0.01
query8 0.03 0.03 0.02
query9 0.52 0.50 0.48
query10 0.54 0.53 0.53
query11 0.11 0.10 0.09
query12 0.12 0.10 0.10
query13 0.61 0.61 0.62
query14 0.79 0.78 0.78
query15 0.79 0.77 0.78
query16 0.37 0.37 0.37
query17 1.02 1.02 1.03
query18 0.24 0.23 0.27
query19 1.91 1.80 1.87
query20 0.01 0.01 0.02
query21 15.47 0.55 0.55
query22 2.02 2.16 1.63
query23 17.13 1.12 1.05
query24 4.23 1.40 1.87
query25 0.37 0.08 0.05
query26 0.66 0.17 0.17
query27 0.05 0.05 0.03
query28 7.60 0.77 0.71
query29 12.59 2.34 2.35
query30 0.55 0.51 0.52
query31 2.80 0.39 0.37
query32 3.39 0.49 0.49
query33 3.05 3.13 3.08
query34 15.26 4.82 4.81
query35 4.90 4.86 4.87
query36 1.06 1.02 1.01
query37 0.06 0.05 0.04
query38 0.03 0.02 0.02
query39 0.02 0.02 0.02
query40 0.15 0.15 0.14
query41 0.06 0.01 0.02
query42 0.02 0.02 0.01
query43 0.02 0.01 0.02
Total cold run time: 102.51 s
Total hot run time: 31.4 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit 4321fae75262e0099fd96d3ff53c3209ff3d11d7 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.4 seconds inserted 10000000 Rows, about 467K ops/s
TPC-H: Total hot run time: 49940 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4321fae75262e0099fd96d3ff53c3209ff3d11d7, data reload: false
------ Round 1 ----------------------------------
q1 17750 4393 4372 4372
q2 2062 154 146 146
q3 10268 1887 1919 1887
q4 10322 1208 1305 1208
q5 8587 3929 3937 3929
q6 236 148 126 126
q7 2038 1618 1594 1594
q8 9930 2745 2724 2724
q9 10664 10447 10234 10234
q10 8674 3495 3525 3495
q11 430 246 247 246
q12 468 306 308 306
q13 18341 3956 4118 3956
q14 358 333 349 333
q15 510 469 462 462
q16 691 568 585 568
q17 1129 972 972 972
q18 7365 6857 7278 6857
q19 1793 1738 1573 1573
q20 554 352 317 317
q21 4467 4166 4179 4166
q22 564 469 485 469
Total cold run time: 117201 ms
Total hot run time: 49940 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4555 4588 4586 4586
q2 361 265 245 245
q3 4292 4255 4227 4227
q4 2772 2787 2778 2778
q5 7202 7173 7065 7065
q6 238 125 121 121
q7 3219 2810 2825 2810
q8 4356 4489 4502 4489
q9 16834 16733 16760 16733
q10 4245 4259 4271 4259
q11 775 713 674 674
q12 1028 864 875 864
q13 7332 3726 3784 3726
q14 453 433 428 428
q15 504 470 459 459
q16 744 693 692 692
q17 3779 3855 3891 3855
q18 8967 8858 8871 8858
q19 1717 1708 1648 1648
q20 2411 2183 2120 2120
q21 8493 8497 8539 8497
q22 1048 945 1002 945
Total cold run time: 85325 ms
Total hot run time: 80079 ms
TPC-DS: Total hot run time: 204699 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4321fae75262e0099fd96d3ff53c3209ff3d11d7, data reload: false
query1 925 423 378 378
query2 6527 2939 2660 2660
query3 6920 200 200 200
query4 20292 18043 17949 17949
query5 19757 6583 6641 6583
query6 294 218 245 218
query7 4169 303 307 303
query8 472 426 464 426
query9 3137 2691 2632 2632
query10 403 308 297 297
query11 11376 10648 10580 10580
query12 121 85 75 75
query13 5584 687 687 687
query14 17962 13654 13168 13168
query15 362 238 256 238
query16 6468 292 271 271
query17 1732 1468 887 887
query18 2291 414 411 411
query19 210 151 147 147
query20 82 79 79 79
query21 190 98 103 98
query22 5089 4942 4893 4893
query23 32496 31929 32027 31929
query24 7014 6560 6542 6542
query25 539 462 413 413
query26 527 162 168 162
query27 1844 305 300 300
query28 6071 2343 2304 2304
query29 2890 2769 2737 2737
query30 241 164 168 164
query31 899 730 740 730
query32 73 64 63 63
query33 405 276 249 249
query34 835 473 468 468
query35 1118 956 942 942
query36 1240 1147 1155 1147
query37 90 62 62 62
query38 3089 2914 2984 2914
query39 1349 1339 1345 1339
query40 204 102 99 99
query41 49 44 45 44
query42 85 84 86 84
query43 742 613 743 613
query44 1161 715 718 715
query45 251 243 239 239
query46 1228 967 961 961
query47 1818 1667 1879 1667
query48 1020 716 722 716
query49 618 380 389 380
query50 855 601 603 601
query51 4701 4609 4684 4609
query52 93 92 82 82
query53 463 334 323 323
query54 2667 2450 2482 2450
query55 87 78 76 76
query56 227 193 217 193
query57 1213 1087 1090 1087
query58 223 204 197 197
query59 4297 3974 4178 3974
query60 213 192 205 192
query61 99 92 95 92
query62 863 501 494 494
query63 486 344 354 344
query64 2505 1523 1511 1511
query65 3613 3584 3597 3584
query66 763 369 373 369
query67 16635 16669 17664 16669
query68 6840 662 651 651
query69 572 368 364 364
query70 1517 1390 1327 1327
query71 395 326 314 314
query72 6553 3526 3507 3507
query73 726 319 319 319
query74 6383 5859 5834 5834
query75 4759 3690 3724 3690
query76 4441 1122 1093 1093
query77 542 251 253 251
query78 12423 11595 11595 11595
query79 6923 661 663 661
query80 2557 398 405 398
query81 524 237 234 234
query82 1338 101 100 100
query83 184 134 138 134
query84 231 70 73 70
query85 1370 322 329 322
query86 364 310 291 291
query87 3277 3026 3046 3026
query88 4801 2327 2305 2305
query89 357 284 302 284
query90 1714 213 213 213
query91 176 144 139 139
query92 58 52 54 52
query93 2267 533 612 533
query94 885 217 212 212
query95 1116 1055 1079 1055
query96 638 327 328 327
query97 6415 6406 6357 6357
query98 188 177 174 174
query99 2947 889 857 857
Total cold run time: 305743 ms
Total hot run time: 204699 ms
ClickBench: Total hot run time: 30.71 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4321fae75262e0099fd96d3ff53c3209ff3d11d7, data reload: false
query1 0.03 0.03 0.02
query2 0.07 0.03 0.02
query3 0.26 0.03 0.05
query4 1.82 0.06 0.07
query5 0.54 0.53 0.52
query6 1.24 0.61 0.63
query7 0.01 0.01 0.01
query8 0.04 0.02 0.03
query9 0.53 0.49 0.48
query10 0.55 0.54 0.54
query11 0.12 0.09 0.10
query12 0.12 0.10 0.09
query13 0.62 0.62 0.60
query14 0.78 0.78 0.81
query15 0.78 0.76 0.77
query16 0.36 0.38 0.36
query17 1.02 0.97 1.00
query18 0.24 0.25 0.25
query19 1.87 1.84 1.88
query20 0.01 0.01 0.01
query21 15.48 0.55 0.57
query22 2.12 2.19 1.78
query23 16.86 0.96 1.07
query24 5.10 1.48 0.91
query25 0.34 0.12 0.05
query26 0.57 0.15 0.15
query27 0.04 0.04 0.04
query28 7.68 0.75 0.76
query29 12.68 2.13 2.22
query30 0.60 0.55 0.51
query31 2.80 0.39 0.38
query32 3.38 0.49 0.50
query33 3.06 3.06 3.06
query34 15.24 4.79 4.80
query35 4.87 4.85 4.84
query36 1.04 1.01 1.01
query37 0.06 0.05 0.04
query38 0.04 0.02 0.02
query39 0.02 0.01 0.01
query40 0.16 0.14 0.14
query41 0.06 0.01 0.02
query42 0.02 0.01 0.01
query43 0.03 0.02 0.02
Total cold run time: 103.26 s
Total hot run time: 30.71 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit 4321fae75262e0099fd96d3ff53c3209ff3d11d7 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.6 seconds inserted 10000000 Rows, about 462K ops/s