lettuce
lettuce copied to clipboard
redis
Investigate for Client-side caching using Redis Cluster and Master/Replica
Feature Request
v6 introduces an awesome new feature (Server-assisted Client-side caching support). There is a limitation currently where it can only be used with Redis Standalone setups. It would be great to extend this support for Redis Cluster and Master/Replica environments.
See https://github.com/lettuce-io/lettuce-core/issues/1281#issuecomment-669028055 for more details.
Is your feature request related to a problem? Please describe
No
Describe the solution you'd like
Client side caching to work with Redis Cluster and Master/Replica.
Describe alternatives you've considered
N/A
Teachability, Documentation, Adoption, Migration Strategy
Ideally it would just use the new CacheFrontend in v6 such that the Redis topology doesn't matter.
Will client-side caching in cluster mode be supported in the future? If true, when is expected?
There are a few challenges to address:
- Re-apply
CLIENT TRACKINGsettings upon reconnect: While that is primary a resiliency feature, in Master/Replica or Cluster mode that is a must since we don't get hold easily of cluster connections. We also need to apply tracking settings when creating a new connection - Topology changes: Whenever the topology changes, we need to adapt to that. Meaning also that we need to carry over tracking settings. More over, how does the migration path look like when in Redis Cluster a slot assignment gets migrated to a new node? What about the server side, does it carry over also all touched keys? I assume not so reconfiguration can cause dropped notifications since the new server is no longer aware of the key subscriptions from the old server. While this could be done (with a lot of effort) on the client side, I think that is too complex.
- Finally, I don't have time to do that in the near future nor I see how I can do that alone in 2021.
Please also refer to #1467 which isn't solved and a pre-requisite to all future work.
I'm also looking forward to this feature. This redis tracking feature will be very helpful in our cache optimization and significantly reduce DB load.
@mp911de I understand that Broadcast won't scale, but as a first step, don't you think it would be great if we have that, obviously as an optional feature though, so that, let the users take call on whether to avail that feature or not based on the benchmarking by the library consumers on use case to use case basis.
Just a thought, we can invalidate the client side cache when topology changes. Would be happy to contribute if there are no potential problems with the idea.
From what I can gather, it seems client side caching in a cluster-topology still isn't supported, correct? Would be a shame if such an awesome feature was held back here; many managed Redis solutions will be cluster setups.
Using Clientside tracking only works if the following assumptions are met:
- Cluster topology never changes (no master/replica failovers): The new master doesn't know about the tracked keys for a connection because a new master will see a new connection.
- Slots reside always on the same cluster nodes (otherwise the new master that handles the moved slot isn't aware of the tracking entries)
And finally
- Finally, I don't have time to do that in the near future nor do I see how I can do that alone in 2022.
holds still true. Trying to shame a maintainer isn't going to help here. Instead, everyone who wants such a feature can take a spike on the feature and start working on a pull request. I'm still not paid for working on Lettuce as I'm maintaining it in my spare time and I do not understand why I should trade a major chunk of my free time for enabling businesses without them giving anything back to OSS or me.
I'm sorry if you felt if my comment was trying to shame you; it was of course not intended that way. I was merely trying to express that this feature (which is awesome in itself) simply lose some if its value if only usable in certain circumstances.
I don't expect anything from open-source projects. It is just a bonus when things like this simply work.
