reactor-netty spring cloud gateway memory leak

Expected behavior No leak

Actual behavior LEAK: ByteBuf.release() was not called before it's garbage-collected

2022-07-14 14:53:03.075 DEBUG   [reactor-http-epoll-3] (Loggers.java:250) - [2f764d98-1, L:/10.xxxxx:xxxx - R:10.xxxxx:xxxx] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
2022-07-14 14:53:14.769 DEBUG   [reactor-http-epoll-3] (Loggers.java:250) - [2f764d98-2, L:/10.xxxxx:xxxx - R:10.xxxxx:xxxx] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver

2022-07-14 14:57:38.536 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	Hint: [2f764d98-2, L:/10.xxxxx:xxxx - R:10.xxxxx:xxxx] Buffered ByteBufHolder in the inbound buffer queue
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:829)
#2:
	Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:829)
Created at:
	io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
	io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
	io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:108)
	io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:359)
	io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:238)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:829)

Steps to reproduce Unfortunately I have no idea how to reproduce it as this error happens randomly in our service in production.

Spring Cloud Gateway version 3.1.1

Reactor Netty version 1.0.16

JVM version (e.g. java -version) 11.0.14

used -Dio.netty.leakDetection.level=PARANOID -Dio.netty.leakDetection.targetRecords=20 _reactor.netty.channel.LeakDetection:debug reactor.netty.channel.FluxReceive:debug

Jul 14 '22 13:07 yanwenliang

Hi @yanwenliang ,

Without having a reproducer scenario for the issue, it is impossible to guess just based on the provided logs. From the logs, we see that the application/framework has subscribed to the response data (see "subscribing inbound receiver "logs), but it does not consume it for whatever reasons.

so, can you elaborate more on the issue and try to setup a reproducible sample project ? can you also upgrade to the latest Reactor Netty version 1.0.21 ? and do you have some exceptions in the logs before the problem happens ?

thanks

Jul 15 '22 09:07 pderop

reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body

reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ HTTP POST "xxxx" [ExceptionHandlingWebHandler]
Original Stack Trace:

2022-07-14 14:53:13.258 ERROR   [reactor-http-epoll-1] (CompositeLog.java:122) - [8ebb749c-2194]  500 Server Error for HTTP POST "xxxx"

reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ HTTP POST "xxxx" [ExceptionHandlingWebHandler]
Original Stack Trace:

reactor.netty.ReactorNetty$InternalNettyException: java.lang.OutOfMemoryError: Direct buffer memory
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.base/java.nio.Bits.reserveMemory(Bits.java:175) ~[na:na]
	at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[na:na]
	at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[na:na]
	at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:649) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:624) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:203) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:187) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:136) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:126) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:394) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) ~[netty-transport-native-unix-common-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

2022-07-14 14:53:08.842 ERROR   [reactor-http-epoll-4] (Loggers.java:315) - Operator called default onErrorDropped

reactor.netty.ReactorNetty$InternalNettyException: java.lang.OutOfMemoryError: Direct buffer memory
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.base/java.nio.Bits.reserveMemory(Bits.java:175) ~[na:na]
	at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[na:na]
	at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[na:na]
	at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:649) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:624) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:203) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:187) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:136) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:126) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:394) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) ~[netty-transport-native-unix-common-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

Jul 18 '22 04:07 yanwenliang

can you also upgrade to the latest Reactor Netty version 1.0.21 ?

Ok, I'll try.

Jul 18 '22 04:07 yanwenliang

@yanwenliang In addition to the recommendation above, please provide more information how SCG is used. Do you use for example some caching filter etc.

Jul 18 '22 06:07 violetagg

I've edited your comment to improve the formatting. You might want to check out this Mastering Markdown guide for future reference.

Jul 18 '22 06:07 violetagg

In addition to the recommendation above, please provide more information how SCG is used. Do you use for example some caching filter etc.

@violetagg This is all filters I used.

filters=@ArrayList[
@OrderedGatewayFilter[[GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyWriteResponseFilter@17c2d509}, order = -1]],
@OrderedGatewayFilter[[GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter@36bf84e}, order = 10000]],
@OrderedGatewayFilter[[GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ReactiveLoadBalancerClientFilter@2c58dcb1}, order = 10150]],
@OrderedGatewayFilter[[GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.LoadBalancerServiceInstanceCookieFilter@680a66dd}, order = 10151]],
@OrderedGatewayFilter[[GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyRoutingFilter@e042c99}, order = 2147483647]]],

Jul 25 '22 06:07 yanwenliang

@yanwenliang Were you able to upgrade? Were you able to prepare a reproducible example?

Jul 29 '22 10:07 violetagg

@yanwenliang I'm closing this one, we can reopen it when you are able to provide more details.

Aug 01 '22 08:08 violetagg

@yanwenliang Were you able to upgrade? Were you able to prepare a reproducible example?

@violetagg I upgrade reactor-netty to 1.0.21, still have memory leak.

Aug 02 '22 07:08 yanwenliang

@yanwenliang Please provide reproducible example.

Aug 02 '22 08:08 violetagg

@yanwenliang I'm closing this one, we can reopen it when you are able to provide more details.

Aug 15 '22 06:08 violetagg

@violetagg @pderop I find the memory leak reason, it has reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation

I add log at org.springframework.http.server.reactive.AbstractServerHttpResponse#writeAndFlushWith

@Override
public final Mono<Void> writeAndFlushWith(Publisher<? extends Publisher<? extends DataBuffer>> body) {
    return new ChannelSendOperator<>(body, inner -> doCommit(() -> writeAndFlushWithInternal(inner)))
        .doOnError(t -> {
            getHeaders().clearContentHeaders();
            log.info(t);
        });
}

this is the log of full request

2022-09-28 11:08:44.608 DEBUG   [reactor-http-epoll-1] (Loggers.java:247) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] New http connection, requesting read
2022-09-28 11:08:44.608 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.left.httpTrafficHandler = reactor.netty.http.server.HttpTrafficHandler), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-09-28 11:08:44.609 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] REGISTERED
2022-09-28 11:08:44.609 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] ACTIVE
2022-09-28 11:08:45.318 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] READ: 399B
2022-09-28 11:08:45.318 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] Increasing pending responses, now 1
2022-09-28 11:08:45.319 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [09938652-1, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@589ed5b8
2022-09-28 11:08:45.320 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc] Created a new pooled channel, now: 148 active connections, 0 inactive connections and 0 pending acquire requests.
2022-09-28 11:08:45.320 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-09-28 11:08:45.320 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc] REGISTERED
2022-09-28 11:08:45.320 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652-1, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] READ COMPLETE
2022-09-28 11:08:46.904 DEBUG   [reactor-http-epoll-1] (Loggers.java:247) - [43a60ddc] Connecting to [/10.106.191.213:22718].
2022-09-28 11:08:46.904 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc] CONNECT: /10.106.191.213:22718
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:247) - [43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] Registering pool release on close event for channel
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] Channel connected, now: 213 active connections, 2 inactive connections and 0 pending acquire requests.
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] ACTIVE
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] onStateChange(PooledConnection{channel=[id: 0x43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718]}, [connected])
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/, connection=PooledConnection{channel=[id: 0x43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718]}}, [configured])
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] Handler is being applied: {uri=http://10.106.191.213:22718/rest/wd/benchmark/response, method=GET}
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718]}}, [request_prepared])
2022-09-28 11:08:47.228 TRACE   [reactor-http-epoll-1] (NettyRoutingFilter.java:143) - outbound route: 43a60ddc, inbound: [09938652-600] 
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [09938652-1, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] FluxReceive{pending=0, cancelled=false, inboundDone=true, inboundError=null}: subscribing inbound receiver
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] WRITE: 616B
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] FLUSH
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718]}}, [request_sent])
2022-09-28 11:08:53.539 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652-1, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] READ COMPLETE
2022-09-28 11:08:53.677 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] READ: 2048B
2022-09-28 11:08:53.677 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] Received response (auto-read:false) : [Content-Type=application/json;charset=UTF-8, Date=Wed, 28 Sep 2022 03:08:47 GMT, content-length=617565]
2022-09-28 11:08:53.677 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718]}}, [response_received])
2022-09-28 11:08:53.677 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
2022-09-28 11:08:53.678 INFO    [reactor-http-epoll-1] (AbstractServerHttpResponse.java:250) - reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation
2022-09-28 11:08:53.681 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] READ: 32768B
2022-09-28 11:08:53.681 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-09-28 11:08:57.509 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652-1, L:/10.xxxxxx:21218 ! R:/10.xxxxxx:47608] INACTIVE
2022-09-28 11:08:57.509 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652, L:/10.xxxxxx:21218 ! R:/10.xxxxxx:47608] UNREGISTERED

Sep 30 '22 11:09 yanwenliang

@yanwenliang Please provide reproducible example.

Sep 30 '22 12:09 violetagg

@yanwenliang Please provide reproducible example.

@violetagg

https://github.com/yanwenliang/scg-leak-demo

start http-server.py
start spring cloud gateway with -Dio.netty.allocator.type=unpooled -Dio.netty.maxDirectMemory=2560000 -Dio.netty.tryReflectionSetAccessible=true
Use curl localhost:8080/scg/leak debug to pause at reactor.netty.http.HttpOperations#send - if (!channel().isActive())

public NettyOutbound send(Publisher<? extends ByteBuf> source) {
    if (!channel().isActive()) {
	    return then(Mono.error(AbortedException.beforeSend()));
    }
    if (source instanceof Mono) {
	    return new PostHeadersNettyOutbound(((Mono<ByteBuf>) source)

evaluate expressionchannel().close()
continue the request to end

and you will find the memory will not release.

Thanks!

Sep 30 '22 19:09 yanwenliang

@violetagg hello, Is there any progress on this issue? Thanks!

Oct 09 '22 03:10 yanwenliang

@yanwenliang ,

just to tell that I'm now checking your reproducer sample project, thanks.

Oct 11 '22 09:10 pderop

Hi @yanwenliang,

I did some investigations on the last reproducer you have provided, but I'm confused with it, first because it uses an older reactor-netty version: 1.0.15 but this issue has been initially reported for a 1.0.16 reactor-netty version.

Also, in the initial post of this issue, the following error has been reported, which seems to indicate that the destination server response has been received, is still in the data structure of reactor-netty, but the subscriber (the gateway ?) has not consuming it:

2022-07-14 14:57:38.536 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	Hint: [2f764d98-2, L:/10.xxxxx:xxxx - R:10.xxxxx:xxxx] Buffered ByteBufHolder in the inbound buffer queue
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:829)

now, the reproducer proposes to add a patch in reactor-netty in order to force all incoming client requests to fail with a "AbortedException: Connection has been closed BEFORE send operation", which is a different use case, and I'm not sure if the patch will (or will not) have a bad side effect (maybe it will cause other problems), and it's a different scenario.

So, sorry to insist but can you please try to reproduce issue without the patch, but with the latest GW version (which is using the reactor-netty 1.0.23 version if I'm correct).

Also, can you enable this logger in TRACE level, it may help to investigate:

org.springframework.cloud.gateway.filter.NettyRoutingFilter

thanks.

Oct 14 '22 08:10 pderop

(I corrected the previous post, please re read it again from here, not from your mail, thank you)

Oct 14 '22 08:10 pderop

Some updates, maybe this is a duplicate of #2114. So, in the initial log, nobody seems to consume the buffer for the request Id 2f764d98-2:

2022-07-14 14:57:38.536 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	Hint: [2f764d98-2, L:/10.xxxxx:xxxx - R:10.xxxxx:xxxx] Buffered ByteBufHolder in the inbound buffer queue
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:829)

So can you check in logs if someone is subscribing to the request Id 2f764d98-2 ? Normally you should find a log similar to this:

[2f764d98-2, L:/10.xxxxx:xxxx - R:app.services/10.xxxxx:xxxx FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver

If you don’t find it, this may explain the leak: no one is subscribing to the 2f764d98-2 request id and the buffer remains in the reactor-netty data structure. So then you need to check why the subscription does not happen.

This issue may be a duplicate of #2114 and #2115, where the root cause may come from this exception (check if you find it from logs):

java.lang.IllegalStateException: Only one connection receive subscriber allowed.
	at reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:182)

please check #2114, and #2115 for more details.

Oct 14 '22 15:10 pderop

in log, about request 2f764d98-2, i just find:

2022-07-14 14:53:14.769 DEBUG   [reactor-http-epoll-3] (Loggers.java:250) - [2f764d98-2, L:/10.xxxxx- R:10.1xxxx/10.xxxx:21784] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver

2022-07-14 14:57:38.536 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	Hint: [2f764d98-2, L:/10.44.90.55:40228 - R:10.102.125.161/10.102.125.161:21784] Buffered ByteBufHolder in the inbound buffer queue
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:829)


2022-07-14 14:57:38.536 ERROR   [parallel-2] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	Hint: [2f764d98-2, L:/10.44.90.55:40228 - R:10.102.125.161/10.102.125.161:21784] Buffered ByteBufHolder in the inbound buffer queue
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:829)


2022-07-14 14:57:38.544 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	Hint: [2f764d98-2, L:/10.44.90.55:40228 - R:10.102.125.161/10.102.125.161:21784] Buffered ByteBufHolder in the inbound buffer queue
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:829)


2022-07-14 14:57:38.560 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	Hint: [2f764d98-2, L:/10.44.90.55:40228 - R:10.102.125.161/10.102.125.161:21784] Buffered ByteBufHolder in the inbound buffer queue
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:829)

2022-07-14 14:57:38.574 ERROR   [reactor-http-epoll-2] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	Hint: [2f764d98-2, L:/10.44.90.55:40228 - R:10.102.125.161/10.102.125.161:21784] Buffered ByteBufHolder in the inbound buffer queue
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:685)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:829)

and there is no log like:

java.lang.IllegalStateException: Only one connection receive subscriber allowed.
	at reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:182)

Oct 14 '22 16:10 yanwenliang

@yanwenliang Can you check whether you have the memory leak if you make this change in Spring Cloud Gateway

https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L124

if (connection != null && connection.channel().isActive()) {

Oct 20 '22 06:10 violetagg

@yanwenliang Can you check whether you have the memory leak if you make this change in Spring Cloud Gateway

https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L124

if (connection != null && connection.channel().isActive()) {

@violetagg Thanks reply! When I change the code, it still has memory leak.

I add log in https://github.com/reactor/reactor-netty/blob/9f839321076e74458343511fa6ade31a0f139408/reactor-netty-http/src/main/java/reactor/netty/http/HttpOperations.java#L115

if (!channel().isActive()) {
        log.info("HttpOperations send is not active {}", channel());
        return then(Mono.error(AbortedException.beforeSend()));
}

And I found this log in my log file:

2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (Loggers.java:271) - HttpOperations send is not active [id: 0xabf65163, L:/10.9xx.xxx.xxx:21428 ! R:/10.2xx.xxx.xxx:50840]

And I add log in https://github.com/spring-projects/spring-framework/blob/5e808ad0183a63944ab08017a13cbc7ed75bc581/spring-web/src/main/java/org/springframework/http/server/reactive/AbstractServerHttpResponse.java#L210

.doOnError(t -> {
    log.info("doOnError: ==============" + t);
    getHeaders().clearContentHeaders();
});

And I also find this log:

2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (AbstractServerHttpResponse.java:247) - doOnError: ==============reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation

The full log is:

2022-10-22 16:32:53.021 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] New http connection, requesting read
2022-10-22 16:32:53.021 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.left.httpTrafficHandler = reactor.netty.http.server.HttpTrafficHandler), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-10-22 16:32:53.021 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] REGISTERED
2022-10-22 16:32:53.021 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] ACTIVE
2022-10-22 16:33:01.256 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] READ: 395B
2022-10-22 16:33:01.255 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 2048, usedDirectMemory: 2055
2022-10-22 16:33:01.256 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] READ: 395B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 47 45 54 20 2f 72 65 73 74 2f 77 64 2f 62 65 6e |GET /rest/wd/ben|
|00000010| 63 68 6d 61 72 6b 2f 72 65 73 70 6f 6e 73 65 20 |chmark/response |
|00000020| 48 54 54 50 2f 31 2e 31 0d 0a 70 74 70 2d 66 6c |HTTP/1.1..ptp-fl|
|00000030| 61 67 3a 20 31 0d 0a 6d 69 6e 54 69 6d 65 3a 20 |ag: 1..minTime: |
|00000040| 30 0d 0a 6d 61 78 54 69 6d 65 3a 20 30 0d 0a 74 |0..maxTime: 0..t|
|00000050| 72 61 63 65 2d 63 6f 6e 74 65 78 74 3a 20 7b 22 |race-context: {"|
|00000060| 6c 61 6e 65 49 64 22 3a 22 22 2c 22 73 74 72 65 |laneId":"","stre|
|00000070| 73 73 54 65 73 74 43 6f 6e 74 65 78 74 22 3a 7b |ssTestContext":{|
|00000080| 22 62 69 7a 4e 61 6d 65 22 3a 22 22 2c 22 76 65 |"bizName":"","ve|
|00000090| 72 73 69 6f 6e 22 3a 30 2c 22 74 65 6d 70 6c 61 |rsion":0,"templa|
|000000a0| 74 65 49 64 22 3a 30 2c 22 72 6f 75 74 65 53 63 |teId":0,"routeSc|
|000000b0| 68 65 64 75 6c 65 54 79 70 65 22 3a 30 2c 22 74 |heduleType":0,"t|
|000000c0| 72 61 66 66 69 63 4d 61 72 6b 73 22 3a 7b 22 36 |rafficMarks":{"6|
|000000d0| 22 3a 7b 22 65 6e 74 69 74 69 65 73 22 3a 5b 7b |":{"entities":[{|
|000000e0| 22 6e 61 6d 65 22 3a 22 70 74 70 46 6c 61 67 22 |"name":"ptpFlag"|
|000000f0| 2c 22 65 78 74 72 61 49 6e 66 6f 73 22 3a 7b 22 |,"extraInfos":{"|
|00000100| 70 74 70 54 61 73 6b 22 3a 22 32 39 34 37 36 35 |ptpTask":"294765|
|00000110| 22 7d 7d 5d 7d 7d 7d 7d 0d 0a 6b 74 72 61 63 65 |"}}]}}}}..ktrace|
|00000120| 44 65 62 75 67 49 64 3a 20 39 39 39 39 34 39 35 |DebugId: 9999495|
|00000130| 35 38 31 30 31 33 0d 0a 58 2d 4b 54 72 61 63 65 |581013..X-KTrace|
|00000140| 2d 49 64 2d 45 6e 61 62 6c 65 64 3a 20 31 0d 0a |-Id-Enabled: 1..|
|00000150| 68 6f 73 74 3a 20 31 30 2e 39 34 2e 31 38 38 2e |host: 10.94.188.|
|00000160| 31 37 32 3a 32 31 34 32 38 0d 0a 61 63 63 65 70 |172:21428..accep|
|00000170| 74 3a 20 2a 2f 2a 0d 0a 75 73 65 72 2d 61 67 65 |t: */*..user-age|
|00000180| 6e 74 3a 20 50 54 50 0d 0a 0d 0a                |nt: PTP....     |
+--------+-------------------------------------------------+----------------+
2022-10-22 16:33:01.256 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 2048, usedDirectMemory: 7
2022-10-22 16:33:01.256 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] Increasing pending responses, now 1
2022-10-22 16:33:01.256 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [abf65163-1, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@5da4b3bf
2022-10-22 16:33:01.257 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f] Created a new pooled channel, now: 0 active connections, 0 inactive connections and 0 pending acquire requests.
2022-10-22 16:33:01.257 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-10-22 16:33:01.257 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f] REGISTERED
2022-10-22 16:33:01.257 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163-1, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] READ COMPLETE
2022-10-22 16:33:02.440 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [5cf2b96f] Connecting to [/10.106.191.213:22718].
2022-10-22 16:33:02.440 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f] CONNECT: /10.106.191.213:22718
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Registering pool release on close event for channel
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Channel connected, now: 35 active connections, 0 inactive connections and 0 pending acquire requests.
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] ACTIVE
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}, [connected])
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [configured])
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Handler is being applied: {uri=http://10.106.191.213:22718/rest/wd/benchmark/response, method=GET}
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [request_prepared])
2022-10-22 16:33:04.145 TRACE   [reactor-http-epoll-1] (NettyRoutingFilter.java:143) - outbound route: 5cf2b96f, inbound: [abf65163-19208]
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [abf65163-1, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] FluxReceive{pending=0, cancelled=false, inboundDone=true, inboundError=null}: subscribing inbound receiver
2022-10-22 16:33:04.145 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 256, usedDirectMemory: 263
2022-10-22 16:33:04.145 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 256, usedDirectMemory: 519
2022-10-22 16:33:04.145 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 512, usedDirectMemory: 1031
2022-10-22 16:33:04.150 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] WRITE: 610B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 47 45 54 20 2f 72 65 73 74 2f 77 64 2f 62 65 6e |GET /rest/wd/ben|
|00000010| 63 68 6d 61 72 6b 2f 72 65 73 70 6f 6e 73 65 20 |chmark/response |
|00000020| 48 54 54 50 2f 31 2e 31 0d 0a 70 74 70 2d 66 6c |HTTP/1.1..ptp-fl|
|00000030| 61 67 3a 20 31 0d 0a 6d 69 6e 54 69 6d 65 3a 20 |ag: 1..minTime: |
|00000040| 30 0d 0a 6d 61 78 54 69 6d 65 3a 20 30 0d 0a 74 |0..maxTime: 0..t|
|00000050| 72 61 63 65 2d 63 6f 6e 74 65 78 74 3a 20 7b 22 |race-context: {"|
|00000060| 6c 61 6e 65 49 64 22 3a 22 22 2c 22 73 74 72 65 |laneId":"","stre|
|00000070| 73 73 54 65 73 74 43 6f 6e 74 65 78 74 22 3a 7b |ssTestContext":{|
|00000080| 22 62 69 7a 4e 61 6d 65 22 3a 22 22 2c 22 76 65 |"bizName":"","ve|
|00000090| 72 73 69 6f 6e 22 3a 30 2c 22 74 65 6d 70 6c 61 |rsion":0,"templa|
|000000a0| 74 65 49 64 22 3a 30 2c 22 72 6f 75 74 65 53 63 |teId":0,"routeSc|
|000000b0| 68 65 64 75 6c 65 54 79 70 65 22 3a 30 2c 22 74 |heduleType":0,"t|
|000000c0| 72 61 66 66 69 63 4d 61 72 6b 73 22 3a 7b 22 36 |rafficMarks":{"6|
|000000d0| 22 3a 7b 22 65 6e 74 69 74 69 65 73 22 3a 5b 7b |":{"entities":[{|
|000000e0| 22 6e 61 6d 65 22 3a 22 70 74 70 46 6c 61 67 22 |"name":"ptpFlag"|
|000000f0| 2c 22 65 78 74 72 61 49 6e 66 6f 73 22 3a 7b 22 |,"extraInfos":{"|
|00000100| 70 74 70 54 61 73 6b 22 3a 22 32 39 34 37 36 35 |ptpTask":"294765|
|00000110| 22 7d 7d 5d 7d 7d 7d 7d 0d 0a 6b 74 72 61 63 65 |"}}]}}}}..ktrace|
|00000120| 44 65 62 75 67 49 64 3a 20 39 39 39 39 34 39 35 |DebugId: 9999495|
|00000130| 35 38 31 30 31 33 0d 0a 58 2d 4b 54 72 61 63 65 |581013..X-KTrace|
|00000140| 2d 49 64 2d 45 6e 61 62 6c 65 64 3a 20 31 0d 0a |-Id-Enabled: 1..|
|00000150| 61 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 75 73 65 |accept: */*..use|
|00000160| 72 2d 61 67 65 6e 74 3a 20 50 54 50 0d 0a 46 6f |r-agent: PTP..Fo|
|00000170| 72 77 61 72 64 65 64 3a 20 70 72 6f 74 6f 3d 68 |rwarded: proto=h|
|00000180| 74 74 70 3b 68 6f 73 74 3d 22 31 30 2e 39 34 2e |ttp;host="10.94.|
|00000190| 31 38 38 2e 31 37 32 3a 32 31 34 32 38 22 3b 66 |188.172:21428";f|
|000001a0| 6f 72 3d 22 31 30 2e 32 38 2e 37 35 2e 31 35 37 |or="10.2xx.xxx.xxx|
|000001b0| 3a 35 30 38 34 30 22 0d 0a 58 2d 46 6f 72 77 61 |:50840"..X-Forwa|
|000001c0| 72 64 65 64 2d 46 6f 72 3a 20 31 30 2e 32 38 2e |rded-For: 10.28.|
|000001d0| 37 35 2e 31 35 37 0d 0a 58 2d 46 6f 72 77 61 72 |75.157..X-Forwar|
|000001e0| 64 65 64 2d 50 72 6f 74 6f 3a 20 68 74 74 70 0d |ded-Proto: http.|
|000001f0| 0a 58 2d 46 6f 72 77 61 72 64 65 64 2d 50 6f 72 |.X-Forwarded-Por|
|00000200| 74 3a 20 32 31 34 32 38 0d 0a 58 2d 46 6f 72 77 |t: 21428..X-Forw|
|00000210| 61 72 64 65 64 2d 48 6f 73 74 3a 20 31 30 2e 39 |arded-Host: 10.9|
|00000220| 34 2e 31 38 38 2e 31 37 32 3a 32 31 34 32 38 0d |4.188.172:21428.|
|00000230| 0a 68 6f 73 74 3a 20 31 30 2e 31 30 36 2e 31 39 |.host: 10.106.19|
|00000240| 31 2e 32 31 33 3a 32 32 37 31 38 0d 0a 63 6f 6e |1.213:22718..con|
|00000250| 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 20 30 0d 0a |tent-length: 0..|
|00000260| 0d 0a                                           |..              |
+--------+-------------------------------------------------+----------------+
2022-10-22 16:33:04.150 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] FLUSH
2022-10-22 16:33:04.151 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 1024, usedDirectMemory: 7
2022-10-22 16:33:04.151 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [request_sent])
2022-10-22 16:33:07.207 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163-1, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] READ COMPLETE
2022-10-22 16:33:07.834 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 2048, usedDirectMemory: 198663
2022-10-22 16:33:07.840 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ: 2048B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 54 54 50 2f 31 2e 31 20 32 30 30 20 0d 0a 43 |HTTP/1.1 200 ..C|
|00000010| 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 70 |ontent-Type: app|
|00000020| 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 |lication/json;ch|
|00000030| 61 72 73 65 74 3d 55 54 46 2d 38 0d 0a 43 6f 6e |arset=UTF-8..Con|
|00000040| 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 31 35 34 |tent-Length: 154|
|00000050| 33 38 38 0d 0a 44 61 74 65 3a 20 53 61 74 2c 20 |388..Date: Sat, |
|00000060| 32 32 20 4f 63 74 20 32 30 32 32 20 30 38 3a 33 |22 Oct 2022 08:3|
|00000070| 33 3a 30 33 20 47 4d 54 0d 0a 0d 0a 7b 22 62 6c |3:03 GMT....{"bl|
|00000080| 6f 63 6b 52 65 73 70 6f 6e 73 65 22 3a 7b 22 72 |ockResponse":{"r|
.............................................................................
|000007b0| 69 7a 65 22 3a 36 37 32 36 2c 22 74 79 70 65 22 |ize":6726,"type"|
|000007c0| 3a 30 2c 22 61 6e 69 6d 61 74 65 64 22 3a 66 61 |:0,"animated":fa|
|000007d0| 6c 73 65 2c 22 74 68 75 6d 62 6e 61 69 6c 49 6d |lse,"thumbnailIm|
|000007e0| 61 67 65 22 3a 7b 22 63 64 6e 55 72 6c 73 22 3a |age":{"cdnUrls":|
|000007f0| 5b 7b 22 75 72 6c 22 3a 22 68 74 74 70 73 3a 2f |[{"url":"https:/|
2022-10-22 16:33:07.841 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Received response (auto-read:false) : [Content-Type=application/json;charset=UTF-8, Date=Sat, 22 Oct 2022 08:33:03 GMT, content-length=154388]
2022-10-22 16:33:07.841 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [response_received])
2022-10-22 16:33:07.841 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (Loggers.java:271) - HttpOperations send is not active [id: 0xabf65163, L:/10.9xx.xxx.xxx:21428 ! R:/10.2xx.xxx.xxx:50840]
2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (AbstractServerHttpResponse.java:247) - doOnError: ==============reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation
2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 2048, usedDirectMemory: 196615
2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 32768, usedDirectMemory: 229383
2022-10-22 16:33:07.936 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ: 32768B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 2f 74 78 2d 66 72 65 65 2d 69 6d 67 73 32 2e 61 |/tx-free-imgs2.a|
|00000010| 63 66 75 6e 2e 63 6e 2f 6b 69 6d 67 2f 62 73 32 |cfun.cn/kimg/bs2|
|00000020| 2f 7a 74 2d 69 6d 61 67 65 2d 68 6f 73 74 2f 43 |/zt-image-host/C|
|00000030| 68 51 77 4f 47 5a 6b 59 54 4a 6c 4e 44 41 79 4d |hQwOGZkYTJlNDAyM|
|00000040| 54 42 68 59 6d 4a 6b 4f 54 51 77 4d 68 43 59 7a |TBhYmJkOTQwMhCYz|
|00000050| 4e 63 76 2e 70 6e 67 22 2c 22 66 72 65 65 54 72 |Ncv.png","freeTr|
|00000060| 61 66 66 69 63 50 72 6f 64 75 63 74 41 62 62 72 |afficProductAbbr|
|00000070| 65 76 69 61 74 69 6f 6e 22 3a 22 22 2c 22 66 72 |eviation":"","fr|
|00000080| 65 65 54 72 61 66 66 69 63 43 64 6e 22 3a 66 61 |eeTrafficCdn":fa|
.............................................................................
|00007f90| 69 6d 67 73 2e 61 69 78 69 66 61 6e 2e 63 6f 6d |imgs.aixifan.com|
|00007fa0| 2f 73 74 79 6c 65 2f 69 6d 61 67 65 2f 32 30 31 |/style/image/201|
|00007fb0| 39 30 37 2f 70 4f 6c 4d 4c 45 66 64 6a 53 31 70 |907/pOlMLEfdjS1p|
|00007fc0| 49 50 69 58 37 74 43 59 6b 56 64 41 69 79 47 56 |IPiX7tCYkVdAiyGV|
|00007fd0| 30 48 79 48 2e 6a 70 67 22 2c 22 76 65 72 69 66 |0HyH.jpg","verif|
|00007fe0| 69 65 64 22 3a 30 2c 22 76 69 64 65 6f 49 64 22 |ied":0,"videoId"|
|00007ff0| 3a 31 30 37 31 33 35 39 38 2c 22 73 75 62 54 69 |:10713598,"subTi|
+--------+-------------------------------------------------+----------------+
2022-10-22 16:33:07.936 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-10-22 16:33:24.833 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163-1, L:/10.9xx.xxx.xxx:21428 ! R:/10.2xx.xxx.xxx:50840] INACTIVE
2022-10-22 16:33:24.833 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163, L:/10.9xx.xxx.xxx:21428 ! R:/10.2xx.xxx.xxx:50840] UNREGISTERED
2022-10-22 16:33:25.663 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 65536, usedDirectMemory: 3014663
2022-10-22 16:33:25.860 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ: 65536B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 74 6c 65 22 3a 22 22 2c 22 6c 69 6e 6b 22 3a 22 |tle":"","link":"|
|00000010| 2f 76 2f 61 63 31 30 39 31 34 36 35 39 22 2c 22 |/v/ac10914659","|
|00000020| 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 e5 8a |description":"..|
|00000030| a9 e6 89 8b ef bc 9a e8 80 81 e6 9d bf ef bc 8c |................|
|00000040| e3 80 8a e9 a3 8e e4 ba 91 e3 80 8b e5 b7 b2 e7 |................|
|00000050| bb 8f e6 88 90 e5 8a 9f e5 8d 87 e7 ba a7 e4 b8 |................|
|00000060| ba e6 a0 bc e6 96 97 e6 b8 b8 e6 88 8f ef bc 8c |................|
|00000070| e7 ac ac e4 b8 80 e8 bd ae e5 b0 8f 42 4f 53 53 |............BOSS|
.............................................................................
|0000ff90| 69 6e 6b 22 3a 22 68 74 74 70 3a 2f 2f 77 77 77 |ink":"http://www|
|0000ffa0| 2e 61 63 66 75 6e 2e 63 6e 2f 73 65 61 72 63 68 |.acfun.cn/search|
|0000ffb0| 2f 3f 23 70 61 67 65 3d 31 3b 71 75 65 72 79 3d |/?#page=1;query=|
|0000ffc0| 25 45 39 25 38 37 25 39 31 25 45 35 25 38 35 25 |%E9%87%91%E5%85%|
|0000ffd0| 38 39 25 45 35 25 42 45 25 41 31 25 45 34 25 42 |89%E5%BE%A1%E4%B|
|0000ffe0| 39 25 39 44 25 45 37 25 39 35 25 38 43 3b 74 79 |9%9D%E7%95%8C;ty|
|0000fff0| 70 65 3d 76 69 64 65 6f 22 2c 22 64 65 73 63 72 |pe=video","descr|
+--------+-------------------------------------------------+----------------+
2022-10-22 16:33:25.860 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:100) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] EXCEPTION: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 65536 byte(s) of direct memory (used: 8323079, max: 8388608)
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 65536 byte(s) of direct memory (used: 8323079, max: 8388608)
        at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:807) ~[classes!/:4.1.75.Final]
        at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:736) ~[classes!/:4.1.75.Final]
        at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:186) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledByteBufAllocator.java:181) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:91) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) ~[netty-transport-native-unix-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [response_completed])
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [disconnecting])
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] CLOSE
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718] Channel closed, now: 579 active connections, 0 inactive connections and 0 pending acquire requests.
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718] USER_EVENT: io.netty.channel.socket.ChannelInputShutdownReadComplete@212f6dea
2022-10-22 16:33:56.662 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718] INACTIVE
2022-10-22 16:33:56.662 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718] onStateChange(PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718]}, [disconnecting])
2022-10-22 16:33:56.662 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718] UNREGISTERED
2022-10-22 16:33:59.533 ERROR   [reactor-http-epoll-1] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        Hint: [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Buffered ByteBufHolder in the inbound buffer queue
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
        reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
        reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
        reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
        reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1567)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1561)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#4:
        Hint: 'reactor.left.loggingHandler' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#5:
        Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
Created at:
        io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
        io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
: 13 leak records were discarded because they were duplicates
: 65534 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.
2022-10-22 16:33:59.583 ERROR   [reactor-http-epoll-1] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        Hint: [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Buffered ByteBufHolder in the inbound buffer queue
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
        reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
        reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
        reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
        reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1567)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1561)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#4:
        Hint: 'reactor.left.loggingHandler' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#5:
        Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
Created at:
        io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
        io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
: 14 leak records were discarded because they were duplicates
: 131081 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

Oct 24 '22 06:10 yanwenliang

@yanwenliang You closed the issue?

Oct 24 '22 06:10 violetagg

@yanwenliang You closed the issue?

sorry，It was a mistake.

Oct 24 '22 06:10 yanwenliang

@violetagg I reopen it, and please see the full log of above, Thanks.

Oct 24 '22 06:10 yanwenliang

@yanwenliang Yes this time the issue is different. I'm going to prepare one PR and will ask you to test it.

Oct 24 '22 06:10 violetagg

@yanwenliang Actually can you try the same file https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L124

if (connection != null) {

Oct 24 '22 07:10 violetagg

@yanwenliang Actually can you try the same file https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L124

if (connection != null) {

@violetagg When I change the code above, the memory leak disappeared.

And the log I added by myself in https://github.com/spring-projects/spring-framework/blob/5e808ad0183a63944ab08017a13cbc7ed75bc581/spring-web/src/main/java/org/springframework/http/server/reactive/AbstractServerHttpResponse.java#L210 is also disappeared.

.doOnError(t -> {
    log.info("doOnError: ==============" + t);
    getHeaders().clearContentHeaders();
});

I can't understand

Oct 24 '22 09:10 yanwenliang

@yanwenliang Can you add logs here

https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L102

https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L69

Oct 24 '22 11:10 violetagg

@yanwenliang Actually can you try the same file https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L124 if (connection != null) {

@violetagg When I change the code above, the memory leak disappeared.

And the log I added by myself in https://github.com/spring-projects/spring-framework/blob/5e808ad0183a63944ab08017a13cbc7ed75bc581/spring-web/src/main/java/org/springframework/http/server/reactive/AbstractServerHttpResponse.java#L210 is also disappeared.
.doOnError(t -> {
    log.info("doOnError: ==============" + t);
    getHeaders().clearContentHeaders();
});
I can't understand

@violetagg Unfortunately, the memory leak appear again

Oct 25 '22 08:10 yanwenliang

reactor-netty reactor-netty copied to clipboard

spring cloud gateway memory leak

reactor-netty
reactor-netty copied to clipboard