Option to reduce connection churn
Hi,
We're using Rqueue 3.2.0 with Redis Cloud (should have available 1000+ connections). Even with Lettuce pooling (20 max-active) we still hit Redis Cloud's hard connection-limit because each poll opens a dedicated connection via openPipeline(). When that happens Lettuce times-out after 10 s and the listener logs. This happen when I have high volume of enqueues (some queues can break down data and enqueue 100x new messages).
I tried to control the number of pool connections with, but no luck. I wonder if there is a way to reuse an open channel?
rqueue:
redis:
database: 0
timeout: 30s
lettuce:
pool:
max-active: 20
max-idle: 10
min-idle: 5
max-wait: 10s
Stack trace
org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.translateException(LettuceConnectionFactory.java:1847) ~[spring-data-redis-3.3.1.jar!/:3.3.1]
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1778) ~[spring-data-redis-3.3.1.jar!/:3.3.1]
at org.springframework.data.redis.connection.lettuce.LettuceConnection.doGetAsyncDedicatedConnection(LettuceConnection.java:968) ~[spring-data-redis-3.3.1.jar!/:3.3.1]
at org.springframework.data.redis.connection.lettuce.LettuceConnection.getOrCreateDedicatedConnection(LettuceConnection.java:1029) ~[spring-data-redis-3.3.1.jar!/:3.3.1]
at org.springframework.data.redis.connection.lettuce.LettuceConnection.openPipeline(LettuceConnection.java:568) ~[spring-data-redis-3.3.1.jar!/:3.3.1]
at jdk.internal.reflect.GeneratedMethodAccessor174.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
at org.springframework.data.redis.core.CloseSuppressingInvocationHandler.invoke(CloseSuppressingInvocationHandler.java:61) ~[spring-data-redis-3.3.1.jar!/:3.3.1]
at jdk.proxy3/jdk.proxy3.$Proxy311.openPipeline(Unknown Source) ~[na:na]
at org.springframework.data.redis.core.RedisTemplate.lambda$executePipelined$1(RedisTemplate.java:472) ~[spring-data-redis-3.3.1.jar!/:3.3.1]
do you've any metrics at server level to identify how many connections it has established? Rqueue does not consume connection properties from the application.properties/yaml file. You can try to create and supply a new connection factory manually.
My theory was that during spikes I enqueue a large amount of messages at once. On spikes I had 2000 messages in my first queue, and those fan out 100x to 200000 of new messages in a short amount of time). And rqueue opened up channels to write each messages but did not closed fast enough to allow us to open. The error rate I received from the timeout disappear when I followed your suggestion and wrote this bean. I can't tell for sure that this was the solution, but the error rate went down from once an hour (on spikes) to zero an hour. It is hard for me to think that Im the first one to encounter this issue, which makes me question my assumptions. My real question is, can we allow enqueueAll that uses the same channel?
This was my solution to limit the connetions:
@Bean
public LettuceConnectionFactory redisConnectionFactory(Environment env) {
String host = env.getProperty("spring.data.redis.host");
int port = Integer.parseInt(env.getProperty("spring.data.redis.port", "0"));
String username = env.getProperty("spring.data.redis.username");
String password = env.getProperty("spring.data.redis.password");
RedisStandaloneConfiguration serverConfig = new RedisStandaloneConfiguration(host, port);
serverConfig.setUsername(username);
serverConfig.setPassword(RedisPassword.of(password));
GenericObjectPoolConfig<?> poolConfig = new GenericObjectPoolConfig<>();
poolConfig.setMaxTotal(20);
poolConfig.setMaxIdle(10);
poolConfig.setMinIdle(5);
poolConfig.setMaxWait(Duration.ofSeconds(10));
LettucePoolingClientConfiguration clientConfig =
LettucePoolingClientConfiguration.builder()
.poolConfig(poolConfig)
.commandTimeout(Duration.ofSeconds(30))
.shutdownTimeout(Duration.ofSeconds(5))
.build();
LettuceConnectionFactory factory = new LettuceConnectionFactory(serverConfig, clientConfig);
factory.setShareNativeConnection(false);
factory.afterPropertiesSet();
return factory;
}
That’s an interesting observation, though I’m not entirely sure what you mean by channel in this context. Could you share a more complete stack trace? That would help in identifying where exactly the failure is occurring whether it's during enqueue, processing, or in the message handler.
It's possible that the solution you applied has temporarily resolved the issue, but if there's an underlying problem with Rqueue, this fix may not hold up under different conditions.
Could you try running a load test to see if the issue reappears, even after setting the pool?