RMapCache Cache expiration not working correctly on a cluster of Redisson clients

[OS-ramamurtisubramanian]

We are using AWS ElastiCache 7.0.7 with cluster mode enabled. We have 2 master and 2 slave nodes. We are connecting adding an expiry listener to RMapCache from multiple microservices in a statefulset in Kubernetes.

Expected behavior We expect all the pods to receive the expired messages.

Actual behavior Only few of the pods receive the expired messages.

Steps to reproduce or test case Add expiry listener to RMapCache with same name across multiple pods and print the expired messages. Push some data in the cache with a timeout. Check if all the pods are getting the expiry events.

Redis version ElastiCache 7.0.7 with clustering

Redisson version 3.26.1

Redisson configuration

[mrniko]

Unable to reproduce. Below is the code example:

            for (int i = 0; i < 10; i++) {
                int j = i;
                RedissonClient rr = Redisson.create(r.getConfig());
                RMapCache<String, String> tt2 = rr.getMapCache("test");
                tt2.addListener(new EntryExpiredListener<String, String>() {

                    @Override
                    public void onExpired(EntryEvent<String, String> event) {
                        System.out.println("expired" + j + " " + event);
                    }
                });
            }

            RMapCache<String, String> tt = r.getMapCache("test");
            tt.addListener(new EntryExpiredListener<String, String>() {

                @Override
                public void onExpired(EntryEvent<String, String> event) {
                    System.out.println("expired " + event);
                }
            });
            tt.put("1", "2", 1, TimeUnit.SECONDS);

            Thread.sleep(10000);

log output:

expired3 org.redisson.api.map.event.EntryEvent@5c4f5ebb
expired4 org.redisson.api.map.event.EntryEvent@7fc2d768
expired7 org.redisson.api.map.event.EntryEvent@57a189ce
expired8 org.redisson.api.map.event.EntryEvent@647f1cab
expired org.redisson.api.map.event.EntryEvent@43753d30
expired5 org.redisson.api.map.event.EntryEvent@307fe2b2
expired6 org.redisson.api.map.event.EntryEvent@5cff5f93
expired9 org.redisson.api.map.event.EntryEvent@de62147
expired1 org.redisson.api.map.event.EntryEvent@5c4f783a
expired2 org.redisson.api.map.event.EntryEvent@1777296f
expired0 org.redisson.api.map.event.EntryEvent@7940476c

[OS-ramamurtisubramanian]

@mrniko I noticed one more thing, With 4 pods I get the following results on redis-cli

PUBSUB SHARDNUMSUB redisson_map_cache_expired:{cacheName} results in 1 PUBSUB NUMSUB redisson_map_cache_expired:{cacheName} results in 3

[OS-ramamurtisubramanian]

@mrniko ,

It seems like the issue is at

https://github.com/redisson/redisson/blob/master/redisson/src/main/java/org/redisson/cluster/ClusterConnectionManager.java#L98

When the Redisson is initialized in multiple JVMs, few of the pods fail to detect whether Sharding is supported and end up subscribing to RedissonTopic instead of RedissonShardedTopic.

I Implmented the RedissonClient interface and copied the required code from Redisson class and I forcefully set the sharding support as true in the SubscribeService and After that it's working as expected.

[mrniko]

Thanks for providing more details. Slave nodes reply with MOVED error. Fixed now.