Warning - java.lang.IllegalStateException: channel not registered to an event loop After upgrade to v4.2.0

[SinghAmbarDT]

Description Recently, I upgrade my SpringCloudGateway service from v4.1.5 to v4.2.0. Also upgraded SpringBoot from v3.3.5 to v3.4.2. Whenever I do deployment my service generates warning

{"timestamp":"2025-02-08T10:37:22.014Z","source":"io.netty.util.concurrent.AbstractEventExecutor","thread":"reactor-http-epoll24", "level":"WARN",
"stack_trace":"java.lang.IllegalStateException: channel not registered to an event loop
	at io.netty.channel.AbstractChannel.eventLoop(AbstractChannel.java:163)
	at io.netty.channel.AbstractChannelHandlerContext.executor(AbstractChannelHandlerContext.java:132)
	at io.netty.channel.AbstractChannelHandlerContext.newPromise(AbstractChannelHandlerContext.java:1011)
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:560)
	at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:906)
	at io.netty.channel.AbstractChannel.close(AbstractChannel.java:243)
	at reactor.netty.http.client.HttpClientOperations.onInboundCancel(HttpClientOperations.java:320)
	at reactor.netty.channel.FluxReceive.disposeAndUnsubscribeReceiver(FluxReceive.java:490)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)",
"message":"A task raised an exception. Task:reactor.netty.channel.FluxReceive$$Lambda/0x00000001005b9c80@77ca8cc7"}

Upgrade Detail

• spring_boot_version= from 3.3.4 to 3.4.2
• spring_cloud_gateway_version= from 4.1.5 to 4.2.0
• spring_dependency_management_version=from 1.1.6 to 1.1.7
• spring_cloud_dependency_version=from 2023.0.3 to 2024.0.0

Other Details

• Using Java 21 (azuljre:21.34.19-alpine-3.20)
• Jib for building Image
• K8s

[spencergibb]

I've edited your comment to improve the formatting. You might want to check out this Mastering Markdown guide for future reference.

Can you describe how to reproduce the problem?

Could you create a complete, minimal, verifiable sample (something that we can unzip attached to this issue or git clone, build, and deploy) that reproduces the problem?

[SinghAmbarDT]

Service is just doing regular predicate based routing to other downstream service and retry filter for HTP 500. Not customisation is done.

As soon as deployment is done. I see these warnings in my code. After rollback everything works normal.

[spencergibb]

Are you able to reproduce it without Kubernetes?

[SinghAmbarDT]

No. Only after deployment to k8s.

[spencergibb]

I'm not sure exactly what gateway has to do with this specifically, then, since it is only reproducible with Kubernetes. The stack trace is purely netty and reactor-netty. Maybe @violetagg might have an idea.

[violetagg]

This seems similar to https://github.com/reactor/reactor-netty/issues/3591, please try the latest Reactore Netty version

[SinghAmbarDT]

Thanks @violetagg. It's been couple of hours and i can't see any event loop warning.

But saw some error. Is it know issue??

LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: Created at:
	io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:144)
	io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
	io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:439)
	io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:320)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Unknown Source)

[violetagg]

@SinghAmbarDT I don't think the last error is related to the original issues.