dogpile.cache
dogpile.cache copied to clipboard
sqlalchemy
feature idea - cache callbacks
Migrated issue, originally created by jvanasco (jvanasco)
before trying to build this out, I wanted to propose these extensions to CacheRegion.get_or_create and CacheRegion.cache_on_arguments:
-
callback_get_fn -
callback_create_fn
If these values are set, before returning the value from the cache it will be passed into the appropriate function as a filter (or a notification).
if GET and callback_get_fn:
value = callback_get_fn(value)
elif CREATE and callback_create_fn:
value = callback_create_fn(value)
return value
In some ways this is similar to a proxy, however this pattern would give a lot more flexibility when the cache generating functions have 'byproducts' of setting up miscellaneous data
Consider this example:
def expensive_function(self):
self.foo = 'bar'
return (1, 2)
def outer(self, args):
@cache_on_arguments
def inner():
(a, b) = self.expensive_function()
return (a, b)
(a, b) = inner(args)
on the first run, self.foo is set; on the second run it is not.
when migrating functions to use dogpile, a lot of issues like this creep up. While the proper route is to avoid this pattern, that can take a lot of rewriting. If there was a mechanism such as a callback that could be used to derive that the value is cached -- then it would be relatively simple to write a few lines of code to ensure parity with whatever environment an uncached data generation would have created.
jvanasco (jvanasco) wrote:
something like this:
def expensive_function(self):
self.foo = 'bar'
return (1, 2)
def _inner_cache_get(self, cached_value):
self.foo = 'bar'
def _inner_cache_create(self, cached_value):
log.debug('cache created')
def outer(self, args):
@cache_on_arguments(callback_get_fn=self._inner_cache_get, callback_create_fn=self._inner_cache_create)
def inner():
(a, b) = self.expensive_function()
return (a, b)
(a, b) = inner(args)
For a bit of context on why i think something like is useful:
I'm caching the output of mako templates (full and partials). For our concerns, the data is cached as a tuple: (html, metadata_dict). The metadata_dict values are needed for various logic concerns and need to be re-integrated back into the request on cache pulls. The re-integration is not a terribly efficient process and not necessary on cache misses (when create is called), making them slower. If we planned on caching responses at the start, this would not be an issue -- but you rarely expect to need to cache something.
Michael Bayer (zzzeek) wrote:
the only event that's not easily trackable is dogpile.core.Lock used the "value getter" but did not use the "value creator", that is, value was in the cache and we didn't call creator. Lock would need to accept a new hook for that. We don't need "callback_create_fn".