Intermittent Issue of "Connection has been closed BEFORE response, while sending request body"

[dukedom26]

We have a few Spring Webflux Application. All of them are using Reactor Netty Core and Reactor Netty Http to create ConnectionProvider and Http Client Respectively to create the WebClient. We are getting intermittently the Connection has been closed BEFORE response, while sending request body error on invoking a call from these services to any downstreams

protected Mono<String> post(String requestPath,
                             String payload,
                             RequestContext context ) {
        return Mono.deferContextual(contextView -> {
            return webClient.build()
                    .post()
                    .uri(uriBuilder -> uriBuilder.path(requestPath).build())
                    .header(AUTHORIZATION, context.get(AUTHORIZATION, String.class))
                    .header(CONTENT_TYPE, APPLICATION_JSON)
                    .bodyValue(payload)
                    .exchangeToMono(clientResponse -> {
                        try (var mdcFrame = LoggingUtils.mdcFrameFromContext(contextView)) {
                            if (clientResponse.statusCode().equals(HttpStatus.OK)) {
                                return clientResponse.bodyToMono(String.class);
                            } else {
                                return Mono.error(new Exception("error"))
                            }
                        }
                    })
                    .retryWhen(Retry.backoff(maxRetry, Duration.ofMillis(waitingTime))
                            .filter(this::isRetryError)
                            .onRetryExhaustedThrow((retryBackoffSpec, retrySignal)
                                    -> new RuntimeException("System error")));
        });
    }

Expected Behavior

The issue shouldn't be happening.

Actual Behavior

The issue is happening quite frequently but not always. Upon retrying, most of the times it works.

2024-10-09 04:48:48.885 [reactor-http-epoll-1] [ERROR] tid=499c2876-e199-45e2-adf3-418848367a3f correlationId= http_request_path=/v1/classify http_query_string=  Exception happened while classification of the document. Exception: org.springframework.web.reactive.function.client.WebClientRequestException: Connection has been closed BEFORE response, while sending request body; nested exception is reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) ~[spring-webflux-5.3.16.jar!/:5.3.16] Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): *__checkpoint ? Request to POST https://org.api.com/document/classify [DefaultWebClient] *__checkpoint ? org.app.filter.CPSWebFilter$$EnhancerBySpringCGLIB$$9a7df50c [DefaultWebFilterChain] *__checkpoint ? org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain] *__checkpoint ? HTTP POST "/v1/classify" [ExceptionHandlingWebHandler] Original Stack Trace: at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) ~[spring-webflux-5.3.16.jar!/:5.3.16] at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55) ~[reactor-core-3.4.31.jar!/:3.4.31] at reactor.core.publisher.Mono.subscribe(Mono.java:4490) ~[reactor-core-3.4.31.jar!/:3.4.31] at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.4.31.jar!/:3.4.31] at 
.....
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar!/:4.1.94.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar!/:4.1.94.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.94.Final.jar!/:4.1.94.Final] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body

Your Environment

• Reactor version(s) used:
• Other relevant libraries versions (eg. netty, ...): reactor-netty-core - 1.0.38, reactor-netty-http - 1.0.38
• JVM version (java -version): OpenJDK Runtime Environment Corretto-11.0.16.8.3 (build 11.0.16+8-LTS)
• OS and version (eg. uname -a): Darwin macos-HGD7JQ02R1 23.5.0 Darwin Kernel Version 23.5.0: Wed May 1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000 arm64

[violetagg]

@dukedom26 Your Reactor Netty version is not supported anymore. Please update to a supported version (more info here)

[violetagg]

Consider checking https://projectreactor.io/docs/netty/release/reference/index.html#faq.connection-closed

[github-actions[bot]]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[github-actions[bot]]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open.