spring-session
spring-session copied to clipboard
spring-projects
Spring Boot / Spring Security App, geting AWS Redis Elasticache production error (failed: Connection reset by peer & CROSSSLOT error)
Getting following connection error with AWS Elasticache (production)
2024-10-19T08:33:26.374Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] f.b.a.r.s.RedisSecurityContextRepository : SAVING AUTHORIZATION REQUEST
2024-10-19T08:33:26.379Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] f.b.a.r.s.RedisSecurityContextRepository : Saved Authorization Request org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@117c1dd in WebSession: org.springframework.security.config.web.server.ServerHttpSecurity$OAuth2LoginSpec$OidcSessionRegistryWebFilter$OidcSessionRegistryServerWebExchange$OidcSessionRegistryWebSession@1420e939
2024-10-19T08:33:26.379Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] f.b.a.r.s.RedisSecurityContextRepository : Authorization Request state: O2Ltdm7yaOIpkBYzAl7k5YY0XktgoAqh3T04kYfIfYI=
2024-10-19T08:33:26.379Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] c.f.b.a.r.OAuth2ServerRedirectStrategy : RUNNING SERVER REDIRECT STRATEGY: 302 FOUND
2024-10-19T08:33:29.682Z INFO 1 --- [BFFApplication] [or-http-epoll-2] .b.a.r.t.CustomServerCsrfTokenRepository : Loading CSRF token
2024-10-19T08:33:29.684Z INFO 1 --- [BFFApplication] [or-http-epoll-2] c.f.b.a.h.c.SPACsrfTokenRequestHandler : Handling CSRF token: MonoSwitchIfEmpty
2024-10-19T08:33:29.684Z INFO 1 --- [BFFApplication] [or-http-epoll-2] .b.a.r.t.CustomServerCsrfTokenRepository : Generating CSRF token
2024-10-19T08:33:29.685Z INFO 1 --- [BFFApplication] [or-http-epoll-2] f.b.a.r.s.RedisSecurityContextRepository : REMOVING AUTHORIZATION REQUEST
2024-10-19T08:33:29.692Z INFO 1 --- [BFFApplication] [xecutorLoop-3-1] f.b.a.r.s.RedisSecurityContextRepository : Removed authorization request
2024-10-19T08:33:29.692Z INFO 1 --- [BFFApplication] [xecutorLoop-3-1] f.b.a.r.s.RedisSecurityContextRepository : Successfully deserialized Authorization Request: org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@705f23d3
io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer
2024-10-19T08:33:29.695Z WARN 1 --- [BFFApplication] [or-http-epoll-1] r.netty.http.client.HttpClientConnect : [04ff363b-2, L:/10.0.141.193:43530 - R:dev-xxxxxxxxxx.auth0.com/104.18.35.70:443] **The connection observed an error, the request cannot be retried as the headers/body were sent**
Caused by: io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer
at java.base/java.lang.Thread.run(Thread.java:831) ~[na:na]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
The only message I see is:
The connection observed an error, the request cannot be retried as the headers/body were sent
If it does pass that stage:
I later also get this 2nd error:
2024-10-19T09:13:07.654Z INFO 1 --- [BFFApplication] [or-http-epoll-2] c.f.b.a.m.GrantedAuthoritiesMapperConfig : PROCESSING AUTHORITY: OIDC_USER
2024-10-19T09:13:07.654Z INFO 1 --- [BFFApplication] [or-http-epoll-2] c.f.b.a.m.GrantedAuthoritiesMapperConfig : OidcUserAuthority ID Token Claims
2024-10-19T09:13:07.660Z INFO 1 --- [BFFApplication] [xecutorLoop-3-5] c.f.b.a.s.CustomSessionIdGenerator : Successfully generated session ID: BFF-1729329187658-c73aad09-7fd2-4980-9fc5-0c2704e97ec4
org.springframework.data.redis.RedisSystemException: Error in execution
Caused by: io.lettuce.core.RedisCommandExecutionException: CROSSSLOT Keys in request don't hash to the same slot
The only message I see here is:
CROSSSLOT Keys in request don't hash to the same slot
Local Laptop Development runs fine (using Redis.com cache)
2024-10-19T09:49:51.937+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] f.b.a.r.s.RedisSecurityContextRepository : SAVING AUTHORIZATION REQUEST
2024-10-19T09:49:51.938+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] c.f.b.a.r.OAuth2ServerRedirectStrategy : RUNNING SERVER REDIRECT STRATEGY: 302 FOUND
2024-10-19T09:49:51.938+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] f.b.a.r.s.RedisSecurityContextRepository : Authorization Request state: 81PGq8kSVgHdOxfqxj_awoELdV5qvFwu50xKuyHSICI=
2024-10-19T09:49:51.938+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] f.b.a.r.s.RedisSecurityContextRepository : Saved Authorization Request org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@14e4bf33 in WebSession: org.springframework.security.config.web.server.ServerHttpSecurity$OAuth2LoginSpec$OidcSessionRegistryWebFilter$OidcSessionRegistryServerWebExchange$OidcSessionRegistryWebSession@6fa54c91
2024-10-19T09:49:55.569+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-2] .b.a.r.t.CustomServerCsrfTokenRepository : Loading CSRF token
2024-10-19T09:49:55.569+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-2] .b.a.r.t.CustomServerCsrfTokenRepository : Generating CSRF token
2024-10-19T09:49:55.570+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-2] c.f.b.a.h.c.SPACsrfTokenRequestHandler : Handling CSRF token: MonoSwitchIfEmpty
2024-10-19T09:49:55.574+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-2] f.b.a.r.s.RedisSecurityContextRepository : REMOVING AUTHORIZATION REQUEST
2024-10-19T09:49:55.610+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-8] f.b.a.r.s.RedisSecurityContextRepository : Successfully deserialized Authorization Request: org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@40a82415
2024-10-19T09:49:55.610+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-8] f.b.a.r.s.RedisSecurityContextRepository : Removed authorization request
2024-10-19T09:49:56.746+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-4] c.f.b.a.m.GrantedAuthoritiesMapperConfig : PROCESSING AUTHORITY: OIDC_USER
2024-10-19T09:49:56.747+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-4] c.f.b.a.m.GrantedAuthoritiesMapperConfig : OidcUserAuthority ID Token Claims
There is an old issue on Github that talks about the 1st error:
https://github.com/reactor/reactor-netty/issues/1774
The 2nd error is due to some Redis Cluster config issue, but I can't find any Spring documentation about it.
Here is my Redis Connection factory config, but I can't see where I am going wrong
@Configuration
internal class RedisConnectionFactoryConfig(
private val springDataProperties: SpringDataProperties,
private val profileProperties: ProfileProperties
) {
// reactive RedisConnectionFactory for key expiration event handling
@Bean
@Primary
fun reactiveRedisConnectionFactory(): ReactiveRedisConnectionFactory {
// configure Redis standalone configuration
val config = RedisStandaloneConfiguration()
config.hostName = springDataProperties.redis.host
config.port = springDataProperties.redis.port
config.setPassword(RedisPassword.of(springDataProperties.redis.password))
// create client options
// Create SSL options if SSL is required
val sslOptions = SslOptions.builder()
.jdkSslProvider() // Or use OpenSslProvider if you prefer
.build()
// Create timeout options
val timeoutOptions = TimeoutOptions.builder()
.fixedTimeout(Duration.ofSeconds(20))
.timeoutCommands(true)
.build()
// cluster specific settings for optimal reliability.
val clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
.enablePeriodicRefresh(Duration.ofSeconds(5))
.dynamicRefreshSources(false)
.adaptiveRefreshTriggersTimeout(Duration.ofSeconds(5))
.enableAllAdaptiveRefreshTriggers().build()
// create socket options
val socketOptions = SocketOptions.builder()
.keepAlive(SocketOptions.DEFAULT_SO_KEEPALIVE)
.tcpNoDelay(SocketOptions.DEFAULT_SO_NO_DELAY)
// time to wait for connection to be established, before considering it as a failed connection
.connectTimeout(Duration.ofSeconds(60))
.build()
val mappingFunction: (HostAndPort) -> HostAndPort = { hostAndPort ->
val host = springDataProperties.redis.host
val addresses: Array<InetAddress> = try {
DnsResolvers.JVM_DEFAULT.resolve(host)
} catch (e: UnknownHostException) {
e.printStackTrace()
emptyArray() // Handle error and return an empty array
}
val cacheIP = addresses.firstOrNull()?.hostAddress ?: ""
var finalAddress = hostAndPort
if (hostAndPort.hostText == cacheIP) {
finalAddress = HostAndPort.of(host, hostAndPort.port)
}
finalAddress
}
val resolver = MappingSocketAddressResolver.create(DnsResolvers.JVM_DEFAULT, mappingFunction)
// customize thread pool size
val clientResources = DefaultClientResources.builder()
.ioThreadPoolSize(8)
.computationThreadPoolSize(8)
.socketAddressResolver(resolver)
.build()
val clusterClientOptionsBuilder = ClusterClientOptions.builder()
.autoReconnect(true)
.pingBeforeActivateConnection(true)
.timeoutOptions(timeoutOptions)
.socketOptions(socketOptions)
.topologyRefreshOptions(clusterTopologyRefreshOptions)
.validateClusterNodeMembership(true)
.suspendReconnectOnProtocolFailure(true)
.disconnectedBehavior(DEFAULT_DISCONNECTED_BEHAVIOR)
.decodeBufferPolicy(DecodeBufferPolicies.ratio(0.5F))
.requestQueueSize(1000)
.maxRedirects(DEFAULT_MAX_REDIRECTS)
.publishOnScheduler(true) //DEFAULT_PUBLISH_ON_SCHEDULER.
.protocolVersion(ProtocolVersion.RESP3) // Use RESP3 Protocol to ensure AUTH command is used for handshake.
// conditionally use sslOptions if profileProperties.active is 'prod'
if (profileProperties.active == "prod") {
clusterClientOptionsBuilder.sslOptions(sslOptions)
}
// build the clientClusterOptions configuration
val clusterClientOptions = clusterClientOptionsBuilder.build()
// configure connection pool settings
fun buildLettucePoolConfig(): GenericObjectPoolConfig<Any> {
val poolConfig = GenericObjectPoolConfig<Any>()
poolConfig.maxTotal = 100
poolConfig.maxIdle = 50
poolConfig.minIdle = 10
poolConfig.setMaxWait(Duration.ofSeconds(120))
poolConfig.timeBetweenEvictionRuns = Duration.ofSeconds(120)
poolConfig.minEvictableIdleTime = Duration.ofMinutes(5)
poolConfig.testOnBorrow = true
poolConfig.testWhileIdle = true
poolConfig.testOnReturn = true
poolConfig.blockWhenExhausted = true
poolConfig.lifo = true
return poolConfig
}
// create Lettuce client configuration with authentication details
val clientConfigBuilder = LettucePoolingClientConfiguration.builder()
// maximum time allowed for a Redis command to execute before the operation is considered timed out.
.commandTimeout(Duration.ofSeconds(60))
.clientResources(clientResources)
.clientOptions(clusterClientOptions)
.poolConfig(buildLettucePoolConfig())
// conditionally enable SSL only if profileProperties.active is 'prod'
if (profileProperties.active == "prod") {
clientConfigBuilder.useSsl()
}
// build the clientConfig configuration
val clientConfig = clientConfigBuilder.build()
// create Lettuce connection factory
return LettuceConnectionFactory(config, clientConfig).apply {
afterPropertiesSet()
validateConnection = false
setShareNativeConnection(true)
}
}
}
Can someone help?
