dubbo icon indicating copy to clipboard operation
dubbo copied to clipboard

EagerThreadPoolExecutor 在Jmeter压测时出现线程池耗尽问题

Open workingonescape opened this issue 3 years ago • 3 comments

最近在学习Dubbo源码,当学习到Dubbo中有关EagerThreadPool相关知识时遇到了一个问题,下面是我的dubbo-demo的一些配置:

image-20220908100611113

image-20220908100900809

然后我使用Jmeter进行压测,模拟高并发场景

image-20220908101057347

image-20220908101040911

紧接着,使用 jmeter -n -t命令执行压测

在这个过程中,会出现 Dubbo线程池耗尽的警告信息

image-20220908101451754

[08/09/22 10:13:07:886 CST] NettyServerWorker-3-4  WARN support.AbortPolicyWithReport:  [DUBBO] Thread pool is EXHAUSTED! Thread Name: dubbo-demo-annotation-provider-198.18.0.1:20880, Pool Size: 10 (active: 1, core: 5, max: 10, largest: 10), Task: 92341 (completed: 92340), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://198.18.0.1:20880!, dubbo version: , current host: 198.18.0.1
[08/09/22 10:13:07:886 CST] NettyServerWorker-3-5  WARN support.AbortPolicyWithReport:  [DUBBO] Thread pool is EXHAUSTED! Thread Name: dubbo-demo-annotation-provider-198.18.0.1:20880, Pool Size: 10 (active: 0, core: 5, max: 10, largest: 10), Task: 92342 (completed: 92341), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://198.18.0.1:20880!, dubbo version: , current host: 198.18.0.1
[08/09/22 10:13:07:886 CST] NettyServerWorker-3-8  WARN support.AbortPolicyWithReport:  [DUBBO] Thread pool is EXHAUSTED! Thread Name: dubbo-demo-annotation-provider-198.18.0.1:20880, Pool Size: 10 (active: 0, core: 5, max: 10, largest: 10), Task: 92342 (completed: 92341), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://198.18.0.1:20880!, dubbo version: , current host: 198.18.0.1
[08/09/22 10:13:07:886 CST] NettyServerWorker-3-9  WARN support.AbortPolicyWithReport:  [DUBBO] Thread pool is EXHAUSTED! Thread Name: dubbo-demo-annotation-provider-198.18.0.1:20880, Pool Size: 10 (active: 0, core: 5, max: 10, largest: 10), Task: 92342 (completed: 92341), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://198.18.0.1:20880!, dubbo version: , current host: 198.18.0.1
[08/09/22 10:13:07:886 CST] NettyServerWorker-3-6  WARN support.AbortPolicyWithReport:  [DUBBO] Thread pool is EXHAUSTED! Thread Name: dubbo-demo-annotation-provider-198.18.0.1:20880, Pool Size: 10 (active: 0, core: 5, max: 10, largest: 10), Task: 92341 (completed: 92340), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://198.18.0.1:20880!, dubbo version: , current host: 198.18.0.1

本来也很正常,Jmeter文件里设置了线程数为 100

image-20220908101657828

但是报错信息的 PoolactiveCount竟然不是10如果线程池耗尽的话这里的activeCount应该是10才对。

image-20220908102122208

这是为什么?

workingonescape avatar Sep 08 '22 02:09 workingonescape

然后我改造了一下,增加了一下日志打印参数,如下:

image-20220908214145171

之后还是使用Jmeter进行压测,结果如下:

image-20220908213619471

submittedTaskCountactivePool Size还是对不上

workingonescape avatar Sep 08 '22 13:09 workingonescape

这是日志 provider.log

workingonescape avatar Sep 08 '22 13:09 workingonescape

image

debug 看了下,active 小于 pool 的原因是中间的 queue 存在竞争以及 worker 获取任务不够及时导致的。queue 队列长度只有 1,在高并发场景下会导致 worker 来不及获取任务就有新任务提交导致被拒绝。

AlbumenJ avatar Sep 15 '22 07:09 AlbumenJ