lettuce icon indicating copy to clipboard operation
lettuce copied to clipboard

Unable to establish a connection to Redis Cluster

Open mjjainmeet opened this issue 1 month ago • 2 comments

Bug Report

I am facing connection issues after upgrading our AWS ElastiCache Redis cluster engine version from 6.2.2 to 7.1. No application-side code changes were made — the only change was the Redis engine version. We are still using the same Redis client version: lettuce-core:5.1.4.RELEASE.

Error logs:

2025-12-08T07:46:02.023Z level=INFO hostIP= correlationId= thread=RMI TCP Connection(5)-192.168.1.44 component=org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] message="Initializing Spring DispatcherServlet 'dispatcherServlet'"
2025-12-08T07:46:03.145Z level=INFO hostIP= correlationId= thread=RMI TCP Connection(4)-192.168.1.44 component=io.lettuce.core.EpollProvider message="Starting without optional epoll library"
2025-12-08T07:46:03.153Z level=INFO hostIP= correlationId= thread=RMI TCP Connection(4)-192.168.1.44 component=io.lettuce.core.KqueueProvider message="Starting without optional kqueue library"
2025-12-08T07:46:03.442Z level=WARN hostIP= correlationId= thread=lettuce-nioEventLoop-6-1 component=io.lettuce.core.cluster.topology.ClusterTopologyRefresh message="Unable to connect to {my_ec_redis_endpoint}:6379"
java.util.concurrent.CompletionException: io.netty.channel.ConnectTimeoutException: connection timed out: {my_ec_redis_endpoint}/{IP}:6379
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
	at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:329)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:263)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:750)
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: {my_ec_redis_endpoint}/{IP}:6379
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261)
	... 10 common frames omitted
2025-12-08T07:46:03.450Z level=WARN hostIP= correlationId= thread=RMI TCP Connection(4)-192.168.1.44 component=org.springframework.boot.actuate.redis.RedisHealthIndicator message="Redis health check failed"
org.springframework.data.redis.connection.PoolException: Could not get a resource from the pool; nested exception is io.lettuce.core.RedisException: Cannot retrieve initial cluster partitions from initial URIs [RedisURI [host='masterdata-ec-cluster-test-v7.u1fzgq.clustercfg.usw2.cache.amazonaws.com', port=6379]]
	at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.getConnection(LettucePoolingConnectionProvider.java:86)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1085)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1065)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getClusterConnection(LettuceConnectionFactory.java:362)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getConnection(LettuceConnectionFactory.java:332)
	at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:132)
	at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:95)
	at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:82)
	at org.springframework.boot.actuate.redis.RedisHealthIndicator.doHealthCheck(RedisHealthIndicator.java:56)
	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:84)
	at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:98)
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:246)
	at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76)
	at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:61)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:126)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:99)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
	at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
	at sun.rmi.transport.Transport$1.run(Transport.java:200)
	at sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: io.lettuce.core.RedisException: Cannot retrieve initial cluster partitions from initial URIs [RedisURI [host='{my_ec_redis_endpoint}', port=6379]]
	at io.lettuce.core.cluster.RedisClusterClient.loadPartitions(RedisClusterClient.java:865)
	at io.lettuce.core.cluster.RedisClusterClient.initializePartitions(RedisClusterClient.java:819)
	at io.lettuce.core.cluster.RedisClusterClient.connect(RedisClusterClient.java:345)
	at org.springframework.data.redis.connection.lettuce.ClusterConnectionProvider.getConnection(ClusterConnectionProvider.java:85)
	at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.lambda$null$0(LettucePoolingConnectionProvider.java:74)
	at io.lettuce.core.support.ConnectionPoolSupport$RedisPooledObjectFactory.create(ConnectionPoolSupport.java:209)
	at io.lettuce.core.support.ConnectionPoolSupport$RedisPooledObjectFactory.create(ConnectionPoolSupport.java:199)
	at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:58)
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:874)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:417)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:346)
	at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:122)
	at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:117)
	at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.getConnection(LettucePoolingConnectionProvider.java:80)
	... 43 common frames omitted
Caused by: io.lettuce.core.RedisConnectionException: Unable to establish a connection to Redis Cluster
	at io.lettuce.core.cluster.topology.AsyncConnections.get(AsyncConnections.java:84)
	at io.lettuce.core.cluster.topology.ClusterTopologyRefresh.loadViews(ClusterTopologyRefresh.java:68)
	at io.lettuce.core.cluster.RedisClusterClient.doLoadPartitions(RedisClusterClient.java:871)
	at io.lettuce.core.cluster.RedisClusterClient.loadPartitions(RedisClusterClient.java:844)
	... 56 common frames omitted
	Suppressed: io.lettuce.core.RedisConnectionException: Unable to connect to {my_ec_redis_endpoint}:6379
		at io.lettuce.core.cluster.topology.ClusterTopologyRefresh.lambda$getConnections$2(ClusterTopologyRefresh.java:231)
		at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
		at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
		at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
		at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:329)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
		at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
		at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
		at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:263)
		at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
		at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164)
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		... 1 common frames omitted
	Caused by: java.util.concurrent.CompletionException: io.netty.channel.ConnectTimeoutException: connection timed out: {my_ec_redis_endpoint}/{IP}:6379
		at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
		at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
		at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
		at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
		... 21 common frames omitted
	Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: {my_ec_redis_endpoint}/{IP}:6379
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261)
		... 10 common frames omitted

Service version details: Java 8 Spring boot 2.1.5.RELEASE Current lettuce lib: lettuce-core-5.1.4.RELEASE

Client creation code:

@Bean("redisConf")
	public RedisClusterConfiguration redisConfiguration() {
		if (properties.getRedisPort() != null) {
			return new RedisClusterConfiguration()
					.clusterNode(properties.getRedisEndpoint(), Integer.valueOf(properties.getRedisPort()));
		}
		return new RedisClusterConfiguration()
				.clusterNode(properties.getRedisEndpoint(), 6379);
	}

	@Bean("redisConfFactory")
	public RedisConnectionFactory redisConnectionFactory(
	        @Qualifier("redisConf") RedisClusterConfiguration redisConfiguration) {
		return new LettuceConnectionFactory(redisConfiguration, createLettuceClientConfig());
	}

	@Bean("redisTemplate")
	public StringRedisTemplate redisTemplate(
	        @Qualifier("redisConfFactory") RedisConnectionFactory factory) {
		return new StringRedisTemplate(factory);
	}

	@Bean
	public RedisHashMapper redisHashMapper(ObjectMapper mapper) {
		return new RedisHashMapper(mapper);
	}

	private LettuceClientConfiguration createLettuceClientConfig() {
		GenericObjectPoolConfig genericObjectPoolConfig = createGenericObjectPoolConfig();
		final SocketOptions socketOptions = SocketOptions.builder()
				.connectTimeout(Duration.ofMillis(MAX_TIMEOUT))
				.build();
		ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
				.enableAllAdaptiveRefreshTriggers()
				.enablePeriodicRefresh(Duration.ofSeconds(TOPOLOGY_REFRESH_INTERVAL))
				.build();

		final ClientOptions clusterClientOptions = ClusterClientOptions.builder()
				.socketOptions(socketOptions)
				.topologyRefreshOptions(topologyRefreshOptions)
				.build();


		return LettucePoolingClientConfiguration.builder()
				.clientOptions(clusterClientOptions)
				.poolConfig(genericObjectPoolConfig)
				.build();
	}

	private GenericObjectPoolConfig createGenericObjectPoolConfig() {
		GenericObjectPoolConfig genericObjectPoolConfig = new GenericObjectPoolConfig();
		genericObjectPoolConfig.setMaxIdle(MAX_IDLE);
		genericObjectPoolConfig.setMaxWaitMillis(MAX_WAIT);
		genericObjectPoolConfig.setMaxTotal(MAX_ACTIVE);
		genericObjectPoolConfig.setMinIdle(MIN_IDLE);

		return genericObjectPoolConfig;
	}

Also, I would like to see proper guidance to resolve the issue.Thanks

mjjainmeet avatar Dec 08 '25 08:12 mjjainmeet

Hey @mjjainmeet

io.lettuce.core.RedisConnectionException: Unable to connect to {my_ec_redis_endpoint}:6379

Seems to me like a connectivity issue - are you able to connect to the given address and port from the machine where lettuce is being deployed?

tishun avatar Dec 08 '25 10:12 tishun

Yes i am able to connect from machine using redis-cli.

mjjainmeet avatar Dec 09 '25 15:12 mjjainmeet

Could you let us know what are the values of the timeout settings you are setting? Could they be too aggressive?

Overall it would be close to impossible for the team to help with connectivity issues, you need to trace the problem on your end, as this is specific to your environment.

tishun avatar Dec 14 '25 13:12 tishun