spring-data-redis
spring-data-redis copied to clipboard
spring-projects
BIG SET keys for every keyspace
Spring data redis version - 2.7.2 Lettuce client version - 6.1.8.RELEASE
When we use CRUDRepository, we can see the keys reference is stored in a separate SET for every keyspace. This is cleared only when the keyspace events are enabled during startup and the SET is cleared via expiration listeners.
When we have multiple instances of the service running, all of them are subscribing to the expiry event and trying to clean up the SET/secondary indexes. Since the keys are added to a SET every-time its saved, there is no way to disable the keyspace events without compromising memory usage or taking external actions like manual flush or a clean up periodically.
If a consumer is not using secondary indexes and do not use findAll or count (for which I believe the SET exists), it would be really great to have an option to disable the separate SET being created for every keyspace. This will help us disable the keyspace events and avoid the additional processing overhead by the service completely.
That's exactly how Redis Repository work. Since Redis has no indexing capabilities, secondary indexes are built externally. Secondary indexes are a core component to make Redis repositories work and they cannot be disabled. Any objects that expire through a TTL require keyspace notifications and a running application for cleanup. That's documented here: https://docs.spring.io/spring-data/redis/docs/current/reference/html/#redis.repositories.expirations
Let us know if there's something else we can support you with.
@mp911de Yes I understand that and agree completely. If someone is using secondary indexes or using findAll or count heavily, this makes perfect sense. My request is if it could be made optional.
if (isNew) { connection.sAdd(toBytes(rdo.getKeyspace()), key); }
If we have no need for a secondary index, we have the potential to completely disable these SET’s at an entity level or at an application level and disable expiration listener. Yes this is a bit of a customisation but imo gives good benefit.
We cannot easily support of customization. For functionality that you want to avoid, I suggest re-implementing the put method of RedisKeyValueAdapter according to your needs.