servicecomb-java-chassis icon indicating copy to clipboard operation
servicecomb-java-chassis copied to clipboard

Published 20 hours ago • verified publisher icon apache

运行一段时间后(5到10天) 会内存溢出

Open huanghezhen opened this issue 2 years ago • 4 comments 

2022-09-13 15:52:07.383  WARN 1 --- [-thread-checker] io.vertx.core.impl.BlockedThreadChecker  : Thread Thread[registry-vert.x-eventloop-thread-2,5,main] has been blocked for 6454 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method) ~[na:na]
        at [email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118) ~[na:na]
        at [email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129) ~[na:na]
        at [email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) ~[na:na]
        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) ~[netty-transport-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:817) ~[netty-transport-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460) ~[netty-transport-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.78.Final.jar:4.1.78.Final]
        at [email protected]/java.lang.Thread.run(Thread.java:833) ~[na:na]
2022-09-13 15:52:07.383  WARN 1 --- [-thread-checker] io.vertx.core.impl.BlockedThreadChecker  : Thread Thread[registry-watch-vert.x-eventloop-thread-1,5,main] has been blocked for 6453 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method) ~[na:na]
        at [email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118) ~[na:na]
        at [email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129) ~[na:na]
        at [email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) ~[na:na]
        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) ~[netty-transport-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:817) ~[netty-transport-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460) ~[netty-transport-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.78.Final.jar:4.1.78.Final]
        at [email protected]/java.lang.Thread.run(Thread.java:833) ~[na:na]
2022-09-13 15:52:08.987  WARN 1 --- [-thread-checker] io.vertx.core.impl.BlockedThreadChecker  : Thread Thread[registry-vert.x-eventloop-thread-1,5,main] has been blocked for 3185 ms, time limit is 2000 ms
2022-09-13 15:52:08.987  WARN 1 --- [-thread-checker] io.vertx.core.impl.BlockedThreadChecker  : Thread Thread[registry-vert.x-eventloop-thread-1,5,main] has been blocked for 4791 ms, time limit is 2000 ms
2022-09-13 15:52:08.989  INFO 1 --- [-center-watch-0] o.a.s.s.task.MicroserviceWatchTask       : microservice vitality-test/order/2.451 UPDATE an instance 540c22dd333311eda2905eb0eeeb1259 status or metadata, [rest://10.100.140.124:5000, highway://10.100.140.124:5001?login=true].
2022-09-13 15:52:11.097  WARN 1 --- [-thread-checker] io.vertx.core.impl.BlockedThreadChecker  : Thread Thread[transport-vert.x-eventloop-thread-1,5,main] has been blocked for 2107 ms, time limit is 2000 ms
2022-09-13 15:52:11.098 ERROR 1 --- [ntloop-thread-1] io.vertx.core.impl.ContextImpl           : Unhandled exception
java.lang.OutOfMemoryError: Java heap space
        at io.vertx.core.buffer.impl.BufferImpl.getBytes(BufferImpl.java:216) ~[vertx-core-4.2.7.jar:4.2.7]
        at org.apache.servicecomb.serviceregistry.client.http.ServiceRegistryClientImpl.lambda$null$1(ServiceRegistryClientImpl.java:198) ~[registry-service-center-2.7.5.jar:2.7.5]
        at org.apache.servicecomb.serviceregistry.client.http.ServiceRegistryClientImpl$$Lambda$1574/0x00000008016213e8.handle(Unknown Source) ~[na:na]
        at io.vertx.core.impl.future.FutureImpl$1.onSuccess(FutureImpl.java:91) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.impl.future.FutureBase$$Lambda$883/0x0000000801333500.run(Unknown Source) ~[na:na]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:81) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:260) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:22) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:79) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.http.impl.HttpClientRequestBase$$Lambda$1570/0x00000008016206f8.handle(Unknown Source) ~[na:na]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:383) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl$$Lambda$1555/0x000000080160b108.handle(Unknown Source) ~[na:na]
        at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:240) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:130) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:610) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.http.impl.Http1xClientConnection$$Lambda$1576/0x00000008016232e0.handle(Unknown Source) ~[na:na]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:71) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:267) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:844) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:716) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:680) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:155) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[vertx-core-4.2.7.jar:4.2.7]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.78.Final.jar:4.1.78.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.78.Final.jar:4.1.78.Final]

应该如何排查呢 使用的是 2.7.5 的版本

huanghezhen avatar Sep 14 '22 07:09 huanghezhen

java.lang.OutOfMemoryError: Java heap space 建议可以先dump内存，分析下内存泄漏的原因。

liubao68 avatar Sep 15 '22 11:09 liubao68

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${BOOT_TKN}.hprof 此类问题，大多发生在业务代码上，框架层面的比较少见，分析一下看看。

trydofor avatar Sep 16 '22 09:09 trydofor

可以试着把Heap size调大点

yanghao605 avatar Sep 17 '22 03:09 yanghao605

好的 我先研究研究 有结果了 会及时反馈的

huanghezhen avatar Sep 17 '22 07:09 huanghezhen