RedisConnection is not released after network loopback

[yxd993293]

trafficstars

Redis version

6.2.5-alpine

Redisson version

3.24.3

Redisson configuration

int totalPoolSize = redisProperties.getJedis().getPool().getMaxActive(); int totalIdleSize = redisProperties.getJedis().getPool().getMaxIdle(); int subscriptionConnectionPoolSize = totalPoolSize / 3; int subscriptionConnectionMinimumIdleSize = totalIdleSize / 3; int masterConnectionPoolSize = totalPoolSize - subscriptionConnectionPoolSize; int masterConnectionMinimumIdleSize = totalIdleSize - subscriptionConnectionMinimumIdleSize; config.useClusterServers() .addNodeAddress(nodes) .setConnectTimeout(timeout) .setPassword(redisProperties.getPassword()) .setMasterConnectionPoolSize(masterConnectionPoolSize) .setMasterConnectionMinimumIdleSize(masterConnectionMinimumIdleSize) .setSlaveConnectionPoolSize(0) .setSlaveConnectionMinimumIdleSize(0) .setRetryAttempts(10) .setSubscriptionConnectionPoolSize(subscriptionConnectionPoolSize) .setSubscriptionConnectionMinimumIdleSize(subscriptionConnectionMinimumIdleSize);

What is the Expected behavior?

No response

What is the Actual behavior?

There is a cluster. Previously, there was a network loop issue. After the network returned to normal, I found that the Redis Sentinel was constantly unhealthy. Upon checking the connection status, I discovered that the number of connections to the Sentinel was very high (10000+), while the maximum configured connections for Redis is 10000.

After investigation, it was found that the number of connections to the Sentinel for each service was quite high. Taking one service as an example, there were 1692 connections to port 18379. When inspecting one of the microservices using Arthas, it was found that there were many unreleased RedisConnection objects.

Among these connections, some were alive but not actual Redis connections.

This indicates an issue with improper connection reclamation, leading to the server reaching its connection limit.

Additional information

No response

[mrniko]

Can you reproduce it with 3.32.0 version? There was a connection leak fix Can you share Redisson logs?

[yxd993293]

Redisson logs: 2024-07-02 09:34:49.570 ERROR [uaa,,] --- [ redisson-netty-2-11] o.r.c.SentinelConnectionManager : org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0xae4c91af, L:/10.110.189.155:37834 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@3ad98c05[Not completed, 2 dependents], params: []

java.util.concurrent.CompletionException: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0xae4c91af, L:/10.110.189.155:37834 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@3ad98c05[Not completed, 2 dependents], params: [] at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1300) at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at org.redisson.client.protocol.CommandData.tryFailure(CommandData.java:87) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:381) at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:205) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:144) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:120) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0xae4c91af, L:/10.110.189.155:37834 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@3ad98c05[Not completed, 2 dependents], params: [] ... 23 common frames omitted

2024-07-02 09:34:49.571 ERROR [uaa,,] --- [ redisson-netty-2-12] o.r.c.SentinelConnectionManager : org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0xa4bdf42a, L:/10.110.189.155:59384 - R:10.110.189.157/10.110.189.157:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@4cfbaeec[Not completed, 2 dependents], params: []

java.util.concurrent.CompletionException: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0xa4bdf42a, L:/10.110.189.155:59384 - R:10.110.189.157/10.110.189.157:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@4cfbaeec[Not completed, 2 dependents], params: [] at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1300) at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at org.redisson.client.protocol.CommandData.tryFailure(CommandData.java:87) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:381) at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:205) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:144) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:120) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0xa4bdf42a, L:/10.110.189.155:59384 - R:10.110.189.157/10.110.189.157:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@4cfbaeec[Not completed, 2 dependents], params: [] ... 23 common frames omitted

2024-07-02 09:34:50.576 ERROR [uaa,,] --- [ redisson-netty-2-19] o.r.c.SentinelConnectionManager : org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x42a7db92, L:/10.110.189.155:37934 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@3ea428c9[Not completed, 2 dependents], params: []

java.util.concurrent.CompletionException: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x42a7db92, L:/10.110.189.155:37934 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@3ea428c9[Not completed, 2 dependents], params: [] at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1300) at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at org.redisson.client.protocol.CommandData.tryFailure(CommandData.java:87) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:381) at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:205) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:144) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:120) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x42a7db92, L:/10.110.189.155:37934 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@3ea428c9[Not completed, 2 dependents], params: [] ... 23 common frames omitted

2024-07-02 09:34:50.577 ERROR [uaa,,] --- [ redisson-netty-2-20] o.r.c.SentinelConnectionManager : org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x89b09f7f, L:/10.110.189.155:59460 - R:10.110.189.157/10.110.189.157:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@26c9e83f[Not completed, 2 dependents], params: []

java.util.concurrent.CompletionException: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x89b09f7f, L:/10.110.189.155:59460 - R:10.110.189.157/10.110.189.157:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@26c9e83f[Not completed, 2 dependents], params: [] at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1300) at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at org.redisson.client.protocol.CommandData.tryFailure(CommandData.java:87) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:381) at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:205) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:144) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:120) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x89b09f7f, L:/10.110.189.155:59460 - R:10.110.189.157/10.110.189.157:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@26c9e83f[Not completed, 2 dependents], params: [] ... 23 common frames omitted

2024-07-02 09:34:51.578 ERROR [uaa,,] --- [ redisson-netty-2-26] o.r.c.SentinelConnectionManager : org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0xe155429b, L:/10.110.189.155:38044 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@2fbb8007[Not completed, 2 dependents], params: []

java.util.concurrent.CompletionException: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0xe155429b, L:/10.110.189.155:38044 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@2fbb8007[Not completed, 2 dependents], params: [] at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1300) at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at org.redisson.client.protocol.CommandData.tryFailure(CommandData.java:87) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:381) at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:205) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:144) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:120) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0xe155429b, L:/10.110.189.155:38044 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@2fbb8007[Not completed, 2 dependents], params: [] ... 23 common frames omitted

2024-07-02 09:34:51.578 ERROR [uaa,,] --- [ redisson-netty-2-27] o.r.c.SentinelConnectionManager : org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x1cce0715, L:/10.110.189.155:60596 - R:10.110.189.157/10.110.189.157:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@611809fa[Not completed, 2 dependents], params: []

java.util.concurrent.CompletionException: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x1cce0715, L:/10.110.189.155:60596 - R:10.110.189.157/10.110.189.157:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@611809fa[Not completed, 2 dependents], params: [] at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1300) at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at org.redisson.client.protocol.CommandData.tryFailure(CommandData.java:87) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:381) at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:205) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:144) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:120) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x1cce0715, L:/10.110.189.155:60596 - R:10.110.189.157/10.110.189.157:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@611809fa[Not completed, 2 dependents], params: [] ... 23 common frames omitted

2024-07-02 09:34:52.365 INFO [uaa,,] --- [ pool-24-thread-7] org.apache.kafka.clients.Metadata : [Consumer clientId=consumer-uaa-3, groupId=uaa] Resetting the last seen epoch of partition mscp-aaa-delete-resource-value-0 to 281 since the associated topicId changed from null to Bo1PRX7LTKuDkR0aeMYv0g 2024-07-02 09:34:52.580 ERROR [uaa,,] --- [ redisson-netty-2-30] o.r.c.SentinelConnectionManager : org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x3728b0ba, L:/10.110.189.155:38152 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@24c478d7[Not completed, 2 dependents], params: []

java.util.concurrent.CompletionException: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x3728b0ba, L:/10.110.189.155:38152 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@24c478d7[Not completed, 2 dependents], params: [] at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1300) at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at org.redisson.client.protocol.CommandData.tryFailure(CommandData.java:87) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:381) at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:205) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:144) at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:120) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.redisson.client.RedisException: ERR max number of clients reached. channel: [id: 0x3728b0ba, L:/10.110.189.155:38152 - R:10.110.189.155/10.110.189.155:18379] command: (PING), promise: java.util.concurrent.CompletableFuture@24c478d7[Not completed, 2 dependents], params: [] ... 23 common frames omitted

[mrniko]

Is this issue reproduceable?

[yxd993293]

We have not attempted to reproduce it yet.

[seakider]

With 1 master, 2 slaves, and 2 sentinels, I used iptables port forwarding to simulate a network loopback, but did not reproduce the situation.

[mrniko]

I'm closing it.