dubbo
dubbo copied to clipboard
EagerThreadPoolExecutor 在Jmeter压测时出现线程池耗尽问题
最近在学习Dubbo源码,当学习到Dubbo中有关EagerThreadPool相关知识时遇到了一个问题,下面是我的dubbo-demo的一些配置:


然后我使用Jmeter进行压测,模拟高并发场景


紧接着,使用 jmeter -n -t命令执行压测
在这个过程中,会出现 Dubbo线程池耗尽的警告信息

[08/09/22 10:13:07:886 CST] NettyServerWorker-3-4 WARN support.AbortPolicyWithReport: [DUBBO] Thread pool is EXHAUSTED! Thread Name: dubbo-demo-annotation-provider-198.18.0.1:20880, Pool Size: 10 (active: 1, core: 5, max: 10, largest: 10), Task: 92341 (completed: 92340), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://198.18.0.1:20880!, dubbo version: , current host: 198.18.0.1
[08/09/22 10:13:07:886 CST] NettyServerWorker-3-5 WARN support.AbortPolicyWithReport: [DUBBO] Thread pool is EXHAUSTED! Thread Name: dubbo-demo-annotation-provider-198.18.0.1:20880, Pool Size: 10 (active: 0, core: 5, max: 10, largest: 10), Task: 92342 (completed: 92341), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://198.18.0.1:20880!, dubbo version: , current host: 198.18.0.1
[08/09/22 10:13:07:886 CST] NettyServerWorker-3-8 WARN support.AbortPolicyWithReport: [DUBBO] Thread pool is EXHAUSTED! Thread Name: dubbo-demo-annotation-provider-198.18.0.1:20880, Pool Size: 10 (active: 0, core: 5, max: 10, largest: 10), Task: 92342 (completed: 92341), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://198.18.0.1:20880!, dubbo version: , current host: 198.18.0.1
[08/09/22 10:13:07:886 CST] NettyServerWorker-3-9 WARN support.AbortPolicyWithReport: [DUBBO] Thread pool is EXHAUSTED! Thread Name: dubbo-demo-annotation-provider-198.18.0.1:20880, Pool Size: 10 (active: 0, core: 5, max: 10, largest: 10), Task: 92342 (completed: 92341), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://198.18.0.1:20880!, dubbo version: , current host: 198.18.0.1
[08/09/22 10:13:07:886 CST] NettyServerWorker-3-6 WARN support.AbortPolicyWithReport: [DUBBO] Thread pool is EXHAUSTED! Thread Name: dubbo-demo-annotation-provider-198.18.0.1:20880, Pool Size: 10 (active: 0, core: 5, max: 10, largest: 10), Task: 92341 (completed: 92340), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://198.18.0.1:20880!, dubbo version: , current host: 198.18.0.1
本来也很正常,Jmeter文件里设置了线程数为 100

但是报错信息的 Pool的activeCount竟然不是10,如果线程池耗尽的话,这里的activeCount应该是10才对。

这是为什么?
然后我改造了一下,增加了一下日志打印参数,如下:

之后还是使用Jmeter进行压测,结果如下:

submittedTaskCount、active和Pool Size还是对不上
这是日志 provider.log

debug 看了下,active 小于 pool 的原因是中间的 queue 存在竞争以及 worker 获取任务不够及时导致的。queue 队列长度只有 1,在高并发场景下会导致 worker 来不及获取任务就有新任务提交导致被拒绝。