dogpile.cache
dogpile.cache copied to clipboard
sqlalchemy
option to drop module/function from key_generator functions in util.py
Migrated issue, originally created by jvanasco (jvanasco)
I tracked down a performance issue to usage of dogpile.cache.region:Region.cache_on_arguments in a few places.
The default key generation is provided key generator functions in dogpile.cache.util, with this relevant code shared by all:
if namespace is None:
namespace = '%s:%s' % (fn.__module__, fn.__name__)
else:
namespace = '%s:%s|%s' % (fn.__module__, fn.__name__, namespace)
My issue was the "base" of the key('%s:%s' % (fn.__module__, fn.__name__)) was incredibly long in a handful of the most-used places, and the keys themselves were taking up a significant chunk of the memory allocated to Redis. Dropping the key length reclaimed a lot of space.
While a key_mangler would solve this, I needed to keep the keys unmangled. The easiest solution was to just reimplement the stock key generator to just use the namespace argument as the key.
This seems like a relatively useful approach for many people, so I wanted to suggest porting it upstream:
I think the relevant changes would be:
- The key_generators have a new kwarg to omit the module+name prefix (like to_str)
- cache_on_arguments(_multi) have a new kwarg for the same, which is just passed to the key_generator (again, like to_str)
Michael Bayer (zzzeek) wrote:
this is...for convenience vs. making your own function_key_generator? the issue of "long keys" is indented to be solved by key_mangler. You can make a key_mangler that just truncates the module/name part, if you dont want to go with making a GUID (I assume that's where your concern wtih mangling is)..
that is, you can get what you want with your own fn_key_geneartor, you can get what you want with a key mangler . where is this lacking?
Michael Bayer (zzzeek) wrote:
how about this:
def function_key_generator(namespace, fn, to_str=compat.string_type, namespace_key_template="%(module)s:%(fn_name)s|%(namespace)s", key_template="%(module)s:%(fn_name)s"):
?
jvanasco (jvanasco) wrote:
re: custom key_generator. This is for convenience. I fixed my deployment in 5 minutes with a custom key_generator, but this seems like it's something other people would want/encounter .
re: mangled guid Yes, having a GUID/hash isn't an option. There are a bunch of tools that dump keys and pull metrics off them.
re: mangled 'other' this particular deployment (web app) already has a key mangler on all regions to handle 'invalid' values. adding splits/regex to the existing region would not work - dogpile is heavily leveraged on some pages (100-1000 hits). splitting it into another "region" with the same settings but a different key_mangler is an option, but would require a lot of work due to how regions are setup, managed, and tested in this app.
the idea for a key_template is much better.