Failed to establish connection to redis cluster mode deployment

[ashok-mariyala]

Expected behavior

Jedis java client should connect with Redis cluster deployment.

Actual behavior

Jedis java client failed to connect with Redis cluster deployment.

Steps to reproduce:

1. Install redis cluster deployment in kubernetes cluster (helm install redis oci://registry-1.docker.io/bitnamicharts/redis-cluster -n redis)
2. Portforward 6379 to local ( kubectl port-forward svc/redis-redis-cluster 6379 -n redis)
3. Execute Jedis java client

Redis / Jedis Configuration

Jedis version: 4.4.3

Redis version:

Chart version: redis-cluster-9.1.3 App Version: 7.2.3

Java version:

java --version
openjdk 11.0.20 2023-07-18
OpenJDK Runtime Environment Temurin-11.0.20+8 (build 11.0.20+8)
OpenJDK 64-Bit Server VM Temurin-11.0.20+8 (build 11.0.20+8, mixed mode)

Java Jedis code snippet

    try {
	var redisHostName = "127.0.0.1"
	var redisPort = 6379;
	var redisUserName = "default";
	var redisPassword = "HhKh5HriPG";
			
	Set<HostAndPort> nodes = new HashSet<>();
        nodes.add(new HostAndPort(redisHostName, redisPort)); 
        cluster = new JedisCluster(nodes, redisUserName, redisPassword);
	        
        cluster.set("testkey", "testvalue");
	System.out.println(cluster.get("testkey"));

	} catch(Exception e) {
		e.printStackTrace();
	}

Exception

redis.clients.jedis.exceptions.JedisClusterOperationException: Cluster retry deadline exceeded.
	at redis.clients.jedis.executors.ClusterCommandExecutor.executeCommand(ClusterCommandExecutor.java:125)
	at redis.clients.jedis.UnifiedJedis.executeCommand(UnifiedJedis.java:183)
	at redis.clients.jedis.UnifiedJedis.set(UnifiedJedis.java:651)
	at com.ashok.redis.RedisClusterCacheClient.initRedisClusterConnection(RedisClusterCacheClient.java:43)
	at com.ashok.redis.RedisClusterCacheClient.<init>(RedisClusterCacheClient.java:28)
	at com.ashok.redis.RedisClusterCacheClient.getInstance(RedisClusterCacheClient.java:54)
	at com.ashok.redis.RedisClusterCacheClient.main(RedisClusterCacheClient.java:23)

[sazzad16]

@ashok-mariyala Can you connect and operate using other clients, e.g. redis-cli?

[ashok-mariyala]

@sazzad16 Thanks for your response. I am able to connect and insert data using redis-cli commands. Please find the following output.

ashok.mariyala@MacBook-Pro-110 ~ % kubectl get pods -n redis
NAME                    READY   STATUS    RESTARTS        AGE
redis-redis-cluster-0   1/1     Running   10 (167m ago)   2d17h
redis-redis-cluster-1   1/1     Running   10 (167m ago)   2d17h
redis-redis-cluster-2   1/1     Running   10 (167m ago)   2d17h
redis-redis-cluster-3   1/1     Running   10 (167m ago)   2d17h
redis-redis-cluster-4   1/1     Running   10              2d17h
redis-redis-cluster-5   1/1     Running   10 (167m ago)   2d17h

ashok.mariyala@MacBook-Pro-110 ~ % kubectl exec -it redis-redis-cluster-0 -n redis -- /bin/bash
I have no name!@redis-redis-cluster-0:/$ redis-cli -c
127.0.0.1:6379> AUTH HhKh5HriPG
OK
127.0.0.1:6379> role
1) "master"
2) (integer) 3836
3) 1) 1) "10.42.0.208"
      2) "6379"
      3) "3836"
127.0.0.1:6379> SET name Ashok
-> Redirected to slot [5798] located at 10.42.0.210:6379
OK
10.42.0.210:6379> GET name
"Ashok"
10.42.0.210:6379> 

[sazzad16]

@ashok-mariyala What happens if you (try to) connect to 10.42.0.208:6379 and 10.42.0.210:6379 directly from Jedis?

[ashok-mariyala]

@sazzad16 I am getting the same error even though i am using 10.42.0.208:6379 and 10.42.0.210:6379 directly from Jedis?

[sazzad16]

@ashok-mariyala You can't get same error (JedisClusterOperationException) from Jedis class/object.

If it was not clear enough, I meant Jedis class/object just this earlier. So try new Jedis("10.42.0.208", 6379) and new Jedis("10.42.0.210", 6379) to connect and execute operations.

[ashok-mariyala]

@sazzad16 Okay sure. I am running redis in Kubernetes and here 10.42.0.208 and 10.42.0.210 addresses are changed to 10.42.0.10 and 10.42.0.252

ubuntu@ashok:~$ kubectl exec -it redis-redis-cluster-0 -n redis -- /bin/bash
I have no name!@redis-redis-cluster-0:/$ redis-cli -c
127.0.0.1:6379> AUTH HhKh5HriPG
OK
127.0.0.1:6379> role
1) "master"
2) (integer) 6188
3) 1) 1) "10.42.0.10"
      2) "6379"
      3) "6188"
127.0.0.1:6379> SET name Ashok
-> Redirected to slot [5798] located at 10.42.0.252:6379
OK
10.42.0.252:6379> GET name
"Ashok"
10.42.0.252:6379>

Here is the result for connecting through jedis. Using 10.42.0.10

try {
	var redisHostName = "10.42.0.10";
	var redisPort = 6379;
	var redisUserName = "default";
	var redisPassword = "HhKh5HriPG";
	jedis = new JedisPooled(redisHostName, redisPort, redisUserName, redisPassword);
		
	jedis.set("Name", "Ashok Kumar");	
	System.out.println(jedis.get("Name"));
} catch(Exception e) {
	e.printStackTrace();
}

Result

redis.clients.jedis.exceptions.JedisMovedDataException: MOVED 8680 10.42.0.252:6379
	at redis.clients.jedis.Protocol.processError(Protocol.java:79)
	at redis.clients.jedis.Protocol.process(Protocol.java:137)
	at redis.clients.jedis.Protocol.read(Protocol.java:192)
	at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:335)
	at redis.clients.jedis.Connection.getOne(Connection.java:317)
	at redis.clients.jedis.Connection.executeCommand(Connection.java:131)
	at redis.clients.jedis.executors.DefaultCommandExecutor.executeCommand(DefaultCommandExecutor.java:24)
	at redis.clients.jedis.UnifiedJedis.executeCommand(UnifiedJedis.java:183)
	at redis.clients.jedis.UnifiedJedis.set(UnifiedJedis.java:651)
	at com.ashok.redis.RedisCacheClient.initRedisConnection(RedisCacheClient.java:60)
	at com.ashok.redis.RedisCacheClient.<init>(RedisCacheClient.java:45)
	at com.ashok.redis.RedisCacheClient.getInstance(RedisCacheClient.java:104)
	at com.ashok.redis.RedisCacheClient.main(RedisCacheClient.java:22)
redis.clients.jedis.exceptions.JedisMovedDataException: MOVED 9411 10.42.0.252:6379
	at redis.clients.jedis.Protocol.processError(Protocol.java:79)
	at redis.clients.jedis.Protocol.process(Protocol.java:137)
	at redis.clients.jedis.Protocol.read(Protocol.java:192)
	at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:335)
	at redis.clients.jedis.Connection.getOne(Connection.java:317)
	at redis.clients.jedis.Connection.executeCommand(Connection.java:131)
	at redis.clients.jedis.executors.DefaultCommandExecutor.executeCommand(DefaultCommandExecutor.java:24)
	at redis.clients.jedis.UnifiedJedis.executeCommand(UnifiedJedis.java:183)
	at redis.clients.jedis.UnifiedJedis.hget(UnifiedJedis.java:1335)
	at com.ashok.redis.RedisCacheClient.get(RedisCacheClient.java:117)
	at com.ashok.redis.RedisCacheClient.main(RedisCacheClient.java:22)

Using 10.42.0.252

try {
	var redisHostName = "10.42.0.252";
	var redisPort = 6379;
	var redisUserName = "default";
	var redisPassword = "HhKh5HriPG";
	jedis = new JedisPooled(redisHostName, redisPort, redisUserName, redisPassword);
		
	jedis.set("Name", "Ashok Kumar");	
	System.out.println(jedis.get("Name"));
} catch(Exception e) {
	e.printStackTrace();
}

Result Ashok Kumar

FYI When i don't use -c flag in the redis-cli tool. I am getting the same error.

I have no name!@redis-redis-cluster-0:/$ redis-cli   
127.0.0.1:6379> AUTH HhKh5HriPG
OK
127.0.0.1:6379> role
1) "master"
2) (integer) 9170
3) 1) 1) "10.42.0.10"
      2) "6379"
      3) "9170"
127.0.0.1:6379> SET name Ashok
(error) MOVED 5798 10.42.0.252:6379
127.0.0.1:6379>

Note We need to use -c flag to insert data in redis cluster mode deployment.

[sazzad16]

There seems not connection related issue. I can only think of increasing maxTotalRetriesDuration parameter. The default is 10 sec.

[ashok-mariyala]

@sazzad16 I tried to increase maxTotalRetriesDuration to 60 and 120 seconds. Still i am facing the same issue. Here is my code block.

try {
	var redisHostName = "127.0.0.1";
	var redisPort = 6379;
	var redisUserName = "default";
	var redisPassword = "HhKh5HriPG";
		
	Set<HostAndPort> nodes = new HashSet<>();
        nodes.add(new HostAndPort(redisHostName, redisPort));
	        
        DefaultJedisClientConfig config = DefaultJedisClientConfig.builder().user(redisUserName).password(redisPassword).build();
	 
        cluster = new JedisCluster(nodes, config, 5, Duration.ofSeconds(120));
	        
        cluster.set("Name", "Ashok Kumar");
        System.out.println(cluster.get("Name"));
} catch(Exception e) {
	e.printStackTrace();
}

Result

redis.clients.jedis.exceptions.JedisClusterOperationException: No more cluster attempts left.
	at redis.clients.jedis.executors.ClusterCommandExecutor.executeCommand(ClusterCommandExecutor.java:129)
	at redis.clients.jedis.UnifiedJedis.executeCommand(UnifiedJedis.java:183)
	at redis.clients.jedis.UnifiedJedis.set(UnifiedJedis.java:651)
	at com.ashok.redis.RedisClusterCacheClient.initRedisClusterConnection(RedisClusterCacheClient.java:50)
	at com.ashok.redis.RedisClusterCacheClient.<init>(RedisClusterCacheClient.java:33)
	at com.ashok.redis.RedisClusterCacheClient.getInstance(RedisClusterCacheClient.java:61)
	at com.ashok.redis.RedisClusterCacheClient.main(RedisClusterCacheClient.java:28)
	Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Failed to connect to any host resolved for DNS name.
		at redis.clients.jedis.DefaultJedisSocketFactory.connectToFirstSuccessfulHost(DefaultJedisSocketFactory.java:63)
		at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:87)
		at redis.clients.jedis.Connection.connect(Connection.java:188)
		at redis.clients.jedis.Connection.initializeFromClientConfig(Connection.java:374)
		at redis.clients.jedis.Connection.<init>(Connection.java:61)
		at redis.clients.jedis.ConnectionFactory.makeObject(ConnectionFactory.java:68)
		at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:571)
		at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:298)
		at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:223)
		at redis.clients.jedis.util.Pool.getResource(Pool.java:38)
		at redis.clients.jedis.ConnectionPool.getResource(ConnectionPool.java:29)
		at redis.clients.jedis.providers.ClusterConnectionProvider.getConnectionFromSlot(ClusterConnectionProvider.java:139)
		at redis.clients.jedis.providers.ClusterConnectionProvider.getConnection(ClusterConnectionProvider.java:95)
		at redis.clients.jedis.executors.ClusterCommandExecutor.executeCommand(ClusterCommandExecutor.java:91)
		... 6 more
		Suppressed: java.net.SocketTimeoutException: Connect timed out
			at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:551)
			at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
			at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
			at java.base/java.net.Socket.connect(Socket.java:633)
			at redis.clients.jedis.DefaultJedisSocketFactory.connectToFirstSuccessfulHost(DefaultJedisSocketFactory.java:73)
			... 19 more

[sazzad16]

@ashok-mariyala Firstly, it's not the same issue.

Seeing only the last error, I would've assumed that it could be port enabling, firewall, or similar issue. But from previous comment, it seems it's not such issue.

Now I'm out of ideas. Sorry, couldn't help you much this time.

[ashok-mariyala]

@sazzad16 Issue is not with the Kubernetes cluster/port enabling/firewall or similar issue. The same cluster is working fine with Jedis client for Redis sentinal and Redis standalone deployments. I am facing issue with only Jedis client for Redis cluster deployment. Could you please let us know who will help on this issue?

Thanks in advance.

[codingSy]

@sazzad16 Issue is not with the Kubernetes cluster/port enabling/firewall or similar issue. The same cluster is working fine with Jedis client for Redis sentinal and Redis standalone deployments. I am facing issue with only Jedis client for Redis cluster deployment. Could you please let us know who will help on this issue?

Thanks in advance.

I encountered the same error, version 4.4.3. There is no obvious exception in the monitoring data from the redis server, but the Jedis client will first execute "redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out", and the error "redis.clients.jedis.exceptions.JedisClusterOperationException: Cluster retryjpg attempts exceeded." will be sent after retrying several times. And I have this problem after trying from version 3.7 to version 4.4.3. I look forward to your reply. Thanks! @ashok-mariyala

[vimal7225s]

Hi @ashok-mariyala @codingSy Have you fixed this issue? I am facing the same error

Error was: Cluster retry deadline exceeded. org.mule.runtime.api.connection.ConnectionException: Cluster retry deadline exceeded. Caused by: redis.clients.jedis.exceptions.JedisClusterOperationException: Cluster retry deadline exceeded. at redis.clients.jedis.executors.ClusterCommandExecutor.executeCommand(ClusterCommandExecutor.java:124) ~[jedis-5.1.1.jar:?]

Here is my code

boolean isSSL = this.tlsContext != null;
if (isSSL) {
	try {
		SSLContext sslContext = this.tlsContext.createSslContext();
		this.sslParameters = sslContext.getDefaultSSLParameters();
		this.sslParameters.setEndpointIdentificationAlgorithm(
				this.endpointIdentificationAlgorithm.name().toLowerCase().replace("disabled", ""));
		this.sslParameters.setCipherSuites(this.tlsContext.getEnabledCipherSuites());
		this.sslParameters.setProtocols(this.tlsContext.getEnabledProtocols());

		this.sslSocketFactory = sslContext.getSocketFactory();
	} catch (Exception e) {
		logger.error("Invalid TLSConfiguration Exception");
	}
}

JedisClientConfig clientConfig = DefaultJedisClientConfig.builder()
		.ssl(isSSL).sslSocketFactory(sslSocketFactory)
		.user("default")
		.password(connectionParams.getPassword())
		.connectionTimeoutMillis(connectionParams.getConnectionTimeout())
		.hostnameVerifier(null)
		.build();

Set<HostAndPort> nodes = new HashSet<HostAndPort>();
nodes.add(new HostAndPort(connectionParams.getHost(), connectionParams.getPort()));

GenericObjectPoolConfig<Connection> poolConfig = new GenericObjectPoolConfig<Connection>();
poolConfig.setMaxTotal(10000);
poolConfig.setMaxIdle(500);

JedisCluster jedisCluster = new JedisCluster(nodes, clientConfig, poolConfig);

[codingSy]

@vimal7225s No, I had no choice but to give up Jedis. After switching to Luttuce, there was no problem.

[AlecBruns]

Any updates on this? I'm getting similar issues with Jedis

[xiaomixin]

Does there any updates for this issue ?