EventBus is not receiving any ReconnectFailedEvent when ConnectionWatchDog has detected a RedisCommandExecutionException.

[zmo123]

Bug Report

Current Behavior

I am using AWS Elasticache Redis instance, I am also using AWS Secret Manager rotation for storing secret for Redis connection, where we have an policy of automatic secret rotating.

I have subscribed to the eventBus provided by Lettuce, I expect to receive a ReconnectFailedEvent if my password has expired or rotated in connection to Redis, I am currently using AWS Elasticache Redis instance for caching purposing.I want to be able to reestablish connection by refreshing the context to build the new lettuceConnectionFactory.

Stack trace

2025-02-04 16:19:48.190 - WARN [catalog-data-services,,,,] 1 --- [oEventLoop-12-3] [] i.l.core.protocol.ConnectionWatchdog : Cannot reconnect to [xxxxxxx]: io.lettuce.core.RedisCommandExecutionException: WRONGPASS invalid username-password pair or user is disabled. java.util.concurrent.CompletionException: io.lettuce.core.RedisCommandExecutionException: WRONGPASS invalid username-password pair or user is disabled. ~~ at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~~ at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~~ at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1141) ~~ at java.base/java.util.concurrent.Comp ---- Caused by: io.lettuce.core.RedisCommandExecutionException: WRONGPASS invalid username-password pair or user is disabled. ~~ at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:147) ~~ at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:116) ~~ ... 31 common frames omitted ~~

Input Code

Input Code

 @Bean
    public void configuration(){
        log.info("RedisConnectionEventListener bean created");
        lettuceConnectionFactory.getClientResources().eventBus().get().subscribe(event -> {
            log.info("RedisConnectionEventListener received event: " + event);
            if (event instanceof ReconnectFailedEvent) {
                LocalTime currentTime = LocalTime.now();

                // Get the current minute of the hour
                int currentMinute = currentTime.getMinute();

                //if current minute is same as previous minute, return
                if(atomicInt.get() == currentMinute){
                    log.info("current ReconnectFailedEvent is being handled at minute:"  + currentMinute);
                    return;
                }

                //update current minute
                atomicInt.set(currentMinute);

                String localAddress = ((ReconnectFailedEvent) event).localAddress().toString();
                String remoteAddress = ((ReconnectFailedEvent) event).remoteAddress().toString();
                log.error("Redis connection failed: " + event + " localAddress:" + localAddress + " RemoteAddress:" + remoteAddress);
                // Handle connection failure
                if(lock.tryLock()) {
                    try {
                        refreshScopeHandler.refreshBeans();
                        scheduler.schedule(() -> {
                            atomicInt.set(-1);
                            log.info("reset RedisConnectionEventListener, current minute: " + currentMinute);
                        }, 1, TimeUnit.MINUTES);
                    } finally {
                        lock.unlock();
                    }
                }else{
                    log.info("Failed to acquire lock");
                }
            }
        });
    }

Expected behavior/code

I expect to receive a ReconnectFailedEvent if my password has expired or rotated in connection to Redis, I am currently using AWS Elasticache Redis instance for caching purposing.I want to be able to reestablish connection by refreshing the context to build the new lettuceConnectionFactory. However no such event is received and I am not able to handle any error related to WRONGPASS.

Environment

• Lettuce version(s): [e.g. 5.0.0.RELEASE, 4.2.2.Final]
• Redis version: [e.g. 4.0.9]

Possible Solution

Additional context

[zmo123]

c.j.p.c.c.RedisConnectionEventListener : RedisConnectionEventListener received event: {}

subsequent event received are shown as blank for some reason?

[zmo123]

hi any help or update is appreciated

[tishun]

Hey @zmo123 ,

the team is busy on some other urgent tasks, I plan to take a look on Friday and give you some feedback

[tishun]

Hey @zmo123 ,

could you also provide a sample of how you set the credentials provider up?

My tests on a simpler setup show that we do get ReconnectFailedEvent and we also have an integration test that tests this. So it must be something in your specific setup that we need to reproduce in order to fix the issue.

Ideally a minimum reproducible example would help speed this up a lot!

[github-actions[bot]]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 2 weeks this issue will be closed.