lettuce
lettuce copied to clipboard
OpsForGeo producing "READONLY You can't write against a read only replica" on READS... only if master & replica configured
Bug Report
This is a strange one and something that is causing us to depend on a single master cache instance for both our reads and our writes.
Current Behavior
When we configure Lettuce with the configuration below, we see it produce the following stacktrace when we perform a read operation using:
redisTemplate.opsForGeo().radius(...)
If we only configure a single master instance using RedisStandaloneConfiguration
we do not see this issue at all.
Stack trace
2021-07-20 13:58:27.904 ERROR 37538 --- [nio-8080-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.data.redis.RedisSystemException: Error in execution; nested exception is io.lettuce.core.RedisCommandExecutionException: READONLY You can't write against a read only replica.] with root cause
io.lettuce.core.RedisCommandExecutionException: READONLY You can't write against a read only replica.
at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:137) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:110) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
at io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:120) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
at io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:111) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:720) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:655) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:572) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
Input Code
Configuration
@Bean
public RedisStaticMasterReplicaConfiguration redisConfiguration() {
RedisStaticMasterReplicaConfiguration redisStaticMasterReplicaConfiguration
= new RedisStaticMasterReplicaConfiguration(properties.getPrimaryEndpoint());
redisStaticMasterReplicaConfiguration.node(properties.getReaderEndpoint());
return redisStaticMasterReplicaConfiguration;
}
@Bean
public LettuceClientConfiguration lettuceClientConfiguration() {
ClientOptions clientOptions = ClientOptions.builder()
.autoReconnect(true)
.build();
return LettuceClientConfiguration.builder()
.readFrom(ReadFrom.REPLICA_PREFERRED)
.clientOptions(clientOptions)
.build();
}
@Bean
public RedisTemplate<String, Object> redisTemplate(RedisStaticMasterReplicaConfiguration redisConfiguration,
LettuceClientConfiguration lettuceClientConfiguration) {
LettuceConnectionFactory lettuceConnectionFactory =
new LettuceConnectionFactory(redisConfiguration, lettuceClientConfiguration);
lettuceConnectionFactory.afterPropertiesSet();
RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();
redisTemplate.setConnectionFactory(lettuceConnectionFactory);
redisTemplate.setEnableTransactionSupport(true);
redisTemplate.setKeySerializer(new StringRedisSerializer());
redisTemplate.setValueSerializer(new JdkSerializationRedisSerializer());
redisTemplate.afterPropertiesSet();
return redisTemplate;
}
Expected behavior/code
GeoResults<RedisGeoCommands.GeoLocation<Object>> result =
redisTemplate.opsForGeo().radius(PREFIX + myKey,
circle);
Should produce an empty/non-empty results object.
Environment
- Lettuce as part of Spring Boot & Data 2.4.2
- AWS - One primary, One read replica
Additional context
If you break point in the code and execute the radius()
command numerous times within the same process, it fails every few times. If, however, you execute it via a REST endpoint, it will fail every time.
This comes from the nature that GEORADIUS
is a command flagged as write-command. For Redis Cluster, Lettuce diverts pure read intentions (variants that do not use STORE
/STOREDIST
) to GEORADIUS_RO
. It would make sense to use a similar approach for Redis Standalone/Replica arrangements.
Output of the COMMAND
command:
1) 1) "georadius_ro"
2) (integer) -6
3) 1) "readonly"
4) (integer) 1
5) (integer) 1
6) (integer) 1
7) 1) "@read"
2) "@geo"
3) "@slow"
173) 1) "georadius"
2) (integer) -6
3) 1) "write"
2) "denyoom"
3) "movablekeys"
4) (integer) 1
5) (integer) 1
6) (integer) 1
7) 1) "@write"
2) "@geo"
3) "@slow"