dogpile.cache
dogpile.cache copied to clipboard
sqlalchemy
Invalidate, Delete, Refresh... using a pattern
Migrated issue, originally created by Diogo Neves (DiogoNeves)
Would be great to invalidate, delete and refresh multiple values using a pattern instead of a list of keys.
I guess this would require changes to the API.
It could be as simple as adding a .keys(pattern) method.
At the moment I've extended Redis (the backend I'm using) and overridden the _multi methods to support a pattern string and lists.
I wouldn't mind giving it a try at implementing this, what do you think?
Michael Bayer (zzzeek) wrote:
how are you generating the keys from this pattern ? if you mean, "filter over all existing keys", that won't work. most backends don't support that. this use case seems more like a recipe to me (we have a recipe section in the docs).
Diogo Neves (DiogoNeves) wrote:
Ok, I wasn't aware of that :S
I'm extending the RedisBackend directly to add the keys() method and access it directly too at the moment.
This is probably not the best approach but I'm just prototyping the system.
Do you have any recommendations? I won't mind updating the documentation ;)
Diogo Neves (DiogoNeves) wrote:
Some code ;)
#!python
class RedisPatternBackend(RedisBackend):
def __init__(self, arguments):
super(RedisPatternBackend, self).__init__(arguments)
def keys(self, pattern):
# TODO: This should play well with the key generator
return self.client.keys(pattern)
Michael Bayer (zzzeek) wrote:
there's a recipe that grabs keys as they are made here: http://dogpilecache.readthedocs.org/en/latest/usage.html#invalidating-a-group-of-related-keys
things like that. there's no keys(), so let's figure out how to track them as they're made locally.
of course if you have arbitrarily large numbers of keys, that won't work.
that's why, overall, plain old time-based invalidation is the easiest and most foolproof system, e.g. http://dogpilecache.readthedocs.org/en/latest/api.html#dogpile.cache.region.CacheRegion.invalidate.
Diogo Neves (DiogoNeves) wrote:
Maybe it's better if I explain the problem ;)
First, the application will be running in multiple workers and the key creation may be done in a different worker from deletion.
For most of the time, the cache behaves like a normal timed cache. The problem is, we have these moments when we need to invalidate all the content related to a user (different types of content, from different methods, called by different parts of the system).
To make it easier, I created a key generator that appends 'uc_{userid}hash' to all user content. Then, when needed, I just delete anything starting with 'uc{userid}'.
Different workers and parts of the system can cache content to speed up read times, but the invalidation is done in one place.
The real problem has a few more details, but that's the bulk of it.
Does it make any sense? Does it help understand how the pattern is useful?
Diogo Neves (DiogoNeves) wrote:
I just realised, re-reading the links you sent, this could be done by keeping a map of user_id -> [hashes associated]... the map itself can be saved in the cache backend ;)
BOOM!
Would that be a good recipe?