Random `Unexpected end of stream` when sending a command

[introom]

Expected behavior

Successfully send the command

Actual behavior

Get Unexpected end of stream exception.

Steps to reproduce:

This scenario happens from time to time. The command sometimes succeeds, and sometimes fails.

The Redis is running along side on my local laptop, so no network issue (conceivably) here.

Jedis jedis = new Jedis();
jedis.set("events/city/rome", "32,15,223,828");

Exception is thrown:

redis.clients.jedis.util.RedisInputStream/ensureFill (RedisInputStream.java:205)
redis.clients.jedis.util.RedisInputStream/readByte (RedisInputStream.java:46)
redis.clients.jedis.Protocol/process (Protocol.java:126)
redis.clients.jedis.Protocol/read (Protocol.java:192)
redis.clients.jedis.Connection/readProtocolWithCheckingBroken (Connection.java:316)
redis.clients.jedis.Connection/getOne (Connection.java:298)
redis.clients.jedis.Connection/executeCommand (Connection.java:123)
redis.clients.jedis.Jedis/set (Jedis.java:4871)
demo/eval13826 (NO_SOURCE_FILE:38)

I used tcpdump and found out no packet is sent to redis from the testing program.

Redis / Jedis Configuration

Jedis version:

4.2.3

Redis version:

7.0.4

Java version:

zulu-17

[introom]

To elaborate more on this. A reproducilbe process is:

1. Create a jedis instance with Jedis jd = new Jedis();
2. Run the command, jd.set("events/city/rome", "32,15,223,828");.

Step 2 fail from time to time.

IF step 2 succeeds, then if I run more jd.set command, they will all succeed

If step 2 fails, if I run more jd.set commands, this time I will got another type of exception

; Execution error (SocketException) at sun.nio.ch.NioSocketImpl/implWrite (NioSocketImpl.java:420).
; Broken pipe
clj꞉demo꞉> 
redis.clients.jedis.Connection/flush (Connection.java:306)
redis.clients.jedis.Connection/getOne (Connection.java:297)
redis.clients.jedis.Connection/executeCommand (Connection.java:123)
redis.clients.jedis.Jedis/set (Jedis.java:4871)

[introom]

Update: I tested with zulu-17.30.15 zulu-18.32.11, the issue keeps appearing.

However, I changed to jdk: zulu-8.52.0.23 zulu-11.54.23, the issue has gone.

Not sure what's happening from the jdk side.

[introom]

Also tested with openjdk-17,same issue.

[yangbodong22011]

@introom Jedis will not be able to recover after encountering an exception (timeout/Redis active disconnection), the recommended correct way is to use JedisPool or JedisPooled.

[dcole-gsn]

Hi, We are encountering the same exception when using JedisPooled.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
	at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:205) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.util.RedisInputStream.readByte(RedisInputStream.java:46) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.Protocol.process(Protocol.java:126) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.Protocol.read(Protocol.java:192) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:316) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:243) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.Connection.ping(Connection.java:400) ~[jedis-4.3.1.jar:?]
	at redis.clients.jedis.ConnectionFactory.validateObject(ConnectionFactory.java:101) ~[jedis-4.3.1.jar:?]
	at org.apache.commons.pool2.impl.GenericObjectPool.evict(GenericObjectPool.java:745) ~[commons-pool2-2.11.1.jar:2.11.1]
	at org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor.run(BaseGenericObjectPool.java:160) ~[commons-pool2-2.11.1.jar:2.11.1]
	at org.apache.commons.pool2.impl.EvictionTimer$WeakRunner.run(EvictionTimer.java:113) ~[commons-pool2-2.11.1.jar:2.11.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_251]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_251]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_251]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_251]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_251]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_251]

In this specific situation the "in" SocketInputStream contains the data '+PONG'. The read fails to return any of the bytes contains within the stream and causes the exception to be raised.

We are using jedis 4.3.1 and have been able to reproduce the same error using 4.2.0.

We create the JedisPooled as

return new JedisPooled(
                getPoolConfig(ConnectionPoolConfig::new),
                hostAndPort.getHost(),
                hostAndPort.getPort(),
                "default",
                password
        );

private <TYPE extends GenericObjectPoolConfig> TYPE getPoolConfig(Supplier<TYPE> constructor) {
        final TYPE poolConfig = constructor.get();
        poolConfig.setMaxTotal(128);
        poolConfig.setMaxIdle(128);
        poolConfig.setMinIdle(16);
        poolConfig.setMaxWait(Duration.ofMillis(500));
        poolConfig.setSoftMinEvictableIdleTime(Duration.ofSeconds(60));
        poolConfig.setTimeBetweenEvictionRuns(Duration.ofSeconds(30));
        poolConfig.setTestOnBorrow(true);
        poolConfig.setTestOnReturn(true);
        poolConfig.setTestWhileIdle(true);
        poolConfig.setNumTestsPerEvictionRun(3);
        poolConfig.setBlockWhenExhausted(true);
        return poolConfig;
    }

Is this observed behaviour expected? I wouldn't expected Jedis to fail reading from the in stream when data is available. I'm sure we are doing something incorrectly, it's just not obvious. Any ideas or suggestion would be appreciated.

Kind regards.

[sazzad16]

Which commons pool version are you using?

Also, a random idea: Have you tested with only one test case instead of three for pooled object?

        poolConfig.setTestOnBorrow(true);
        poolConfig.setTestOnReturn(true);
        poolConfig.setTestWhileIdle(true);

Even better, disabling all these tests?

[bxb100]

@introom Did you run Jedis object in multi-thread?

[imsuperman0920]

Hi, I got the same exception when using JedisCluster. I connected the redis cluster using JedisCluster and created a pipeline object by jedisCluster.pipelined(). When I got multi keys with pipeline , the exception happened.

Java 11 Jedis 4.4.1 Redis 6.2.6

Kind regards.

[nijaaouikhalil]

Hello, I am having the same behavior for openjdk8 JedisPool final JedisPoolConfig conf = new JedisPoolConfig(); conf.setMaxTotal(100); conf.setTestOnBorrow(false); conf.setTestOnReturn(false); conf.setTestOnCreate(false); conf.setTestWhileIdle(false); conf.setMinEvictableIdleTimeMillis(60000); conf.setTimeBetweenEvictionRunsMillis(30000); conf.setNumTestsPerEvictionRun(-1); conf.setFairness(true);

@Bean
public JReJSON redisJson(JedisPool jedisPool) {

	return new JReJSON(jedisPool);
}

it works on localhost but not when using secure connection, PS: I see in the server that the document gets create but the exception gets thrown

[sazzad16]

@nijaaouikhalil What is your socket timeout? If it's small, you may consider increasing it.

[nijaaouikhalil]

here is the order of my beans: private static JedisPoolConfig initPoolConfig() {

	final JedisPoolConfig conf = new JedisPoolConfig();
	conf.setMaxTotal(100);
	conf.setTestOnBorrow(false);
	conf.setTestOnReturn(false);
	conf.setTestOnCreate(false);
	conf.setTestWhileIdle(false);
	conf.setMinEvictableIdleTimeMillis(60000);
	conf.setTimeBetweenEvictionRunsMillis(30000);
	conf.setNumTestsPerEvictionRun(-1);
	conf.setFairness(true);
	return conf;
}
@Bean
public JedisPool jedisPool() {

	if (ENV_WITHOUT_DB_PZWD_VALUE.equals(this.runningEnv)) {
		return new JedisPool(initPoolConfig(), serverHost, serverPort, socketTimeoutMillis, null);
	} else {
		return new JedisPool(initPoolConfig(), serverHost, serverPort, socketTimeoutMillis,
				username, decode(this.encodedPassword), true);
	}
}

@Bean
public RedisConnectionFactory redisConnectionFactory(JedisPool jedisPool) {
	RedisStandaloneConfiguration configuration = new RedisStandaloneConfiguration();

	JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory(configuration);

	jedisConnectionFactory.afterPropertiesSet();
	return jedisConnectionFactory;
}
@Bean
public JReJSON redisJson(JedisPool jedisPool) {
	return new JReJSON(jedisPool);
}

@Bean
public RedisTemplate<?, ?> redisTemplate(RedisConnectionFactory redisConnectionFactory) {
	RedisTemplate<?, ?> template = new RedisTemplate<>();
	template.setConnectionFactory(redisConnectionFactory);

	// Configure Redis JSON Serializer/Deserializer if needed
	return template;
}

if I set configuration.setHostName(serverHost); // Set your server host configuration.setPort(serverPort); // Set your server port in the redisConnectionFactory the application fails and I get unexpected end of stream,

locally it works when I simply don't pass them in the RedisConnectionFactory but when I host it it doesn't work and the RedisConnectionFactory still tries to connect to the localhost:6379, how to force it to use the existing jedispool?