spring-cloud-gateway recvAddress(..) failed: Connection reset by peer

Question while reading request from nginx get connection reset by peer error.The frequency of this error is very low, about one in 100,000 SCG:10.224.0.74:8080,10.233.54.43:8080(10.233.54.43 is cluster ip,10.224.0.74 is pod ip) NGINX:10.224.2.210:43972 SCG version: <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-gateway-server</artifactId> 3.1.6 SCG log: 2023-11-10 13:31:04.874[DEBUG][reactor-http-epoll-25][trace_id=2776c77d7d917db222dfa4b119a32c93][span_id=a1163ea1fbb47826][reactor.netty.http.server.HttpServerOperations][5009a976, L:/10.224.0.74:8080 ! R:/10.224.2.210:43972] Failed flushing last frame Original Stack Trace: *__checkpoint ⇢ HTTP POST "/openapi/v1/user/update-user" [ExceptionHandlingWebHandler] *__checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain] Error has been observed at the following site(s): Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer 2023-11-10 13:31:04.873[ERROR][reactor-http-epoll-25][trace_id=2776c77d7d917db222dfa4b119a32c93][span_id=a1163ea1fbb47826][reactor.core.publisher.Operators]Operator called default onErrorDropped 2023-11-10 13:31:04.873[DEBUG][reactor-http-epoll-25][trace_id=2776c77d7d917db222dfa4b119a32c93][span_id=a1163ea1fbb47826][reactor.netty.channel.ChannelOperations][5009a976-1, L:/10.224.0.74:8080 ! R:/10.224.2.210:43972] [HttpServer] Channel inbound receiver cancelled (channel disconnected). io.netty.channel.unix.Errors$NativeIoException: sendAddress(..) failed: Broken pipe 2023-11-10 13:31:04.872[DEBUG][reactor-http-epoll-25][trace_id=2776c77d7d917db222dfa4b119a32c93][span_id=a1163ea1fbb47826][reactor.netty.http.server.HttpServerOperations][5009a976-1, L:/10.224.0.74:8080 ! R:/10.224.2.210:43972] Sending last HTTP packet was not successful, terminating the channel 2023-11-10 13:31:04.872[DEBUG][reactor-http-epoll-25][trace_id=2776c77d7d917db222dfa4b119a32c93][span_id=a1163ea1fbb47826][reactor.netty.http.server.HttpServerOperations][5009a976-1, L:/10.224.0.74:8080 - R:/10.224.2.210:43972] Decreasing pending responses, now 0 2023-11-10 13:31:04.871[DEBUG][reactor-http-epoll-25][trace_id=2776c77d7d917db222dfa4b119a32c93][span_id=a1163ea1fbb47826][reactor.netty.http.server.HttpServerOperations][5009a976-1, L:/10.224.0.74:8080 - R:/10.224.2.210:43972] No sendHeaders() called before complete, sending zero-length header 2023-11-10 13:31:04.871[DEBUG][reactor-http-epoll-25][trace_id=2776c77d7d917db222dfa4b119a32c93][span_id=a1163ea1fbb47826][reactor.netty.http.server.HttpServerOperations][5009a976-1, L:/10.224.0.74:8080 - R:/10.224.2.210:43972] Last HTTP response frame Original Stack Trace: *__checkpoint ⇢ HTTP POST "/openapi/v1/user/update-user" [ExceptionHandlingWebHandler] *__checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain] Error has been observed at the following site(s): Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer 2023-11-10 13:31:04.868[DEBUG][reactor-http-epoll-25][trace_id=2776c77d7d917db222dfa4b119a32c93][span_id=a1163ea1fbb47826][reactor.netty.channel.FluxReceive][5009a976-1, L:/10.224.0.74:8080 - R:/10.224.2.210:43972] [terminated=false, cancelled=false, pending=1, error=null]: subscribing inbound receiver 2023-11-10 13:31:04.866[INFO][reactor-http-epoll-25][trace_id=2776c77d7d917db222dfa4b119a32c93][span_id=a1163ea1fbb47826][SERVICE_GATEWAY]Request Info [ method=POST;url=/openapi/v1/user/update-user ] 2023-11-10 13:31:04.865[DEBUG][reactor-http-epoll-25][trace_id=2776c77d7d917db222dfa4b119a32c93][span_id=a1163ea1fbb47826][reactor.netty.http.server.HttpServer][5009a976-1, L:/10.224.0.74:8080 - R:/10.224.2.210:43972] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@6708ebfd 2023-11-10 13:31:04.865[DEBUG][reactor-http-epoll-25][trace_id=2776c77d7d917db222dfa4b119a32c93][span_id=a1163ea1fbb47826][reactor.netty.http.server.HttpServerOperations][5009a976, L:/10.224.0.74:8080 - R:/10.224.2.210:43972] Increasing pending responses, now 1 2023-11-10 13:31:04.865[DEBUG][reactor-http-epoll-25][trace_id=][span_id=][reactor.netty.transport.TransportConfig][5009a976, L:/10.224.0.74:8080 - R:/10.224.2.210:43972] Initialized pipeline DefaultChannelPipeline{(reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (io.opentelemetry.javaagent.shaded.instrumentation.netty.v4_1.internal.server.HttpServerTracingHandler = io.opentelemetry.javaagent.shaded.instrumentation.netty.v4_1.internal.server.HttpServerTracingHandler), (reactor.left.httpTrafficHandler = reactor.netty.http.server.HttpTrafficHandler), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)} 2023-11-10 13:31:04.865[DEBUG][reactor-http-epoll-25][trace_id=][span_id=][reactor.netty.http.server.HttpServerOperations][5009a976, L:/10.224.0.74:8080 - R:/10.224.2.210:43972] New http connection, requesting read Nginx log: 2023/11/10 05:31:04 [error] 7#7: *5159863 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.224.0.0, server: _, request: "POST /openapi/v1/user/update-user HTTP/1.1", upstream: "http://10.233.54.43:8080/openapi/v1/user/update-user", host: "xxxxxxxx" 2023/11/10 05:31:04 [error] 7#7: *5159863 open() "/usr/share/nginx/html/50x.html" failed (2: No such file or directory), client: 10.224.0.0, server: _, request: "POST /openapi/v1/user/update-user HTTP/1.1", upstream: "http://10.233.54.43:8080/openapi/v1/user/update-user", host: "xxxxxxxx"

Nov 10 '23 07:11 huiyangcode

I also encountered the same problem. Do you have any good solutions?

Dec 19 '23 07:12 jhyhhmail

Hello , I am also seeing something similar within our application. This starter occurring after migrating from 2.7.12.

Spring boot 3.2.0 Spring Cloud 2023.0.0 SCG 4.1.0 SCG Server 4.1.0

Feb 06 '24 22:02 williejay2017

@williejay2017 how often do you see it?

Mar 11 '24 22:03 spencergibb

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Mar 18 '24 22:03 spring-cloud-issues

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

Mar 25 '24 22:03 spring-cloud-issues