spring-cloud-gateway
spring-cloud-gateway copied to clipboard
spring-cloud
AbortedException: Connection has been closed BEFORE send operation
Spring Cloud Gateway throws hundreds of "reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation" exceptions everyday.
The version of spring boot and cloud:
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.8.RELEASE</version>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>Hoxton.SR10</version>
The config of spring cloud gateway:
server.undertow.io-threads = 1 server.undertow.worker-threads = 1 hystrix.threadpool.default.coreSize = 10 hystrix.threadpool.default.maxQueueSize = 100 spring.cloud.consul.discovery.healthCheckInterval = 5s feign.httpclient.max-connections = 200 spring.cloud.gateway.httpclient.pool.max-idle-time = 20s spring.cloud.gateway.httpclient.pool.max-life-time = 120s spring.cloud.gateway.httpclient.pool.max-connections = 1000 spring.cloud.gateway.httpclient.pool.acquire-timeout = 5000 spring.cloud.gateway.httpclient.pool.type = fixed spring.cloud.gateway.httpclient.connect-timeout = 3000 spring.cloud.gateway.httpclient.response-timeout = 30s
The config of the remote peer: feign.httpclient.max-connections-per-route = 150 server.connection-timeout = 20000
The error log:
content:2021-04-20 09:55:58,125 [iot-gateway,,] [PID:2572] [reactor-http-epoll-3] ERROR r.n.channel.ChannelOperationsHandler - Error was received while reading the incoming data. The connection will be closed.
reactor.core.Exceptions$BubblingException: reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation
at reactor.core.Exceptions.bubble(Exceptions.java:173)
at reactor.core.publisher.Operators.onErrorDropped(Operators.java:635)
at com.alibaba.csp.sentinel.adapter.reactor.InheritableBaseSubscriber.onError(InheritableBaseSubscriber.java:169)
at com.alibaba.csp.sentinel.adapter.reactor.SentinelReactorSubscriber.onError(SentinelReactorSubscriber.java:37)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:97)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:97)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:97)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:97)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:97)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:227)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:97)
at reactor.core.publisher.Operators$MonoSubscriber.onError(Operators.java:1834)
at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onError(MonoIgnoreThen.java:306)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:97)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:97)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:251)
at org.springframework.http.server.reactive.ChannelSendOperator$WriteCompletionBarrier.onError(ChannelSendOperator.java:414)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:97)
at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onError(MonoIgnoreElements.java:76)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:97)
at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onError(Operators.java:2034)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:97)
at reactor.core.publisher.Operators$MonoSubscriber.onError(Operators.java:1834)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreInner.onError(MonoIgnoreThen.java:235)
at reactor.core.publisher.Operators.error(Operators.java:196)
at reactor.core.publisher.MonoError.subscribe(MonoError.java:52)
at reactor.core.publisher.Mono.subscribe(Mono.java:4252)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172)
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
at reactor.core.publisher.Mono.subscribe(Mono.java:4252)
at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:207)
at reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:80)
at reactor.core.publisher.Mono.subscribe(Mono.java:4252)
at org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onNext(ChannelSendOperator.java:192)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:90)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:90)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:90)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:265)
at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:371)
at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:358)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:677)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation
at reactor.netty.channel.AbortedException.beforeSend(AbortedException.java:60)
at reactor.netty.http.HttpOperations.then(HttpOperations.java:166)
at reactor.netty.ReactorNetty$OutboundThen.
@hooverhe
AbortedException: Connection has been closed BEFORE send operation means that we were able to obtain the connection from the pool and the connection was alive but before sending the request the remote peer closed the connection.
Is there an elegant way to reduce log level or catch this exception? At the moment it spams our logs with Errors that we can not prevent, allowing users to replicate Errors in our logs by closing Browser-Tabs while Uploading etc. Since we do have alarms in monitoring watching for exceptions in log files, this is a problem for us. We would like to handle this exception and log it ourselves or reduce log level.
@violetagg
AbortedException: Connection has been closed BEFORE send operationmeans that we were able to obtain the connection from the pool and the connection was alive but before sending the request the remote peer closed the connection.
The connection of the WebClient or of the client to the Netty application server? Any tip on how to solve this problem? We have several apps with this same behavior at @mercadolibre
@deploydesexta it is about connection2 in the diagram below i.e. the connection established between WebClient and the target service
client -> Spring Cloud Gateway -> target service
^ ^
connection1 connection2
Please check FAQ for possible solutions.
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.
@deploydesexta In case this is still an issue for you, or anyone else on this thread.
I noticed this error arising on my system when accidentally broadcasting messages to an expired session. This may be the case for you as well. After fixing my state, I no longer see such messages.
@ztayluh thanks for the reply! By expired session you mean what session? And what state you fixed? Can you share more? I didn't get it where to look up to fix 😞
Sorry for any confusion! In my case, I was holding onto the WebsocketSessions in a cache and broadcasting a WebsocketMessage to the respective sessions.
private var activeSessions = ConcurrentHashMap<String, Flux<WebSocketSession>>()
If I disconnected my client, I was not properly clearing out the WebsocketSession. So, when a new message was received I would try to broadcast to my clients. That error message would occur when trying to send to invalid sessions in my cache.
You may be needing to capture a closeStatus() to properly remove the session, like so:
val socketClosure = session.closeStatus() .doOnNext { removeSession(session, channel) }
I hope this helps!