python-musicbrainzngs icon indicating copy to clipboard operation
python-musicbrainzngs copied to clipboard

Implement a caching mechanism to minimize the number of HTTP requests sent by the client

Open agateblue opened this issue 8 years ago • 4 comments

In a project of mine, I end up doing a lot of requests to the public Musicbrainz API endpoint. Due to the throttle policies in place (or the heavy load that sometimes occur on this public instance), some of my requests end up raising a 503 error.

To partially adress that, I've implemented a basic cache mechanism, to store API results and return the API results from the cache when possible, effectively minimizing the number of HTTP requests made.

I'll let you have a look at the commits for more details, but the implementation is a follows:

  1. If a cache is configured, the _mb_request function will first try to retrieve the value from the cache
  2. If not present, the actual request is sent to the API
  3. When the API returns, the response is stored in the cache for further use

This implementation is agnostic regarding how the cache behave. A simple cache (DictCache) is provided to demonstrate the cache behaviour, but more complex caches (using real cache systems, such as Redis or Memcached) are totally doable.

The cache key for each response is built using the URL and arguments (such as includes), so there should not be any collision.

The build may fail on Travis, because the tests I wrote require the mock library. I've added it to the tox dependencies, but since it's not called on Travis, I'm not sure how to make the build install the library before running tests.

Documentation is attached in the commit, if you want more details.

Any feedback is welcome, there are probably a few things to fix before this can be merged :)

agateblue avatar Jun 18 '16 19:06 agateblue

This is a great patch, thanks! I probably won't have time to review it until August, but I'm in favour of including something like this.

alastair avatar Jun 22 '16 09:06 alastair

No worries, we're all pretty busy. Feel free to tell me if there is anything I can do to speed up / help with the reviewing process.

agateblue avatar Jun 22 '16 13:06 agateblue

Hi Eliot and others, if we happen to merge #199 before this, you can find a branch where I've already merged this on top of #199 and fixed the conflicts at https://github.com/mineo/python-musicbrainz-ngs/tree/request-reloaded-testmerge.

mineo avatar Jul 01 '16 16:07 mineo

Hey @mineo, thank you for your hard work on this, I'm really happy to hear things are moving forward !

agateblue avatar Jul 03 '16 15:07 agateblue