Performance of `api/v2/exercisebaseinfo/?limit=900`

[bbkz]

Use case

In my setup, when i open /exercise/overview/ it takes 2-3 minutes to load the content of api/v2/exercisebaseinfo/?limit=900. I run it on raspberry's and similar, so it may just be a hardware limitation.

Workaround

So i came up with the following workaround. I set EXERCISE_CACHE_TTL to 25 hours and run a cronjob every 24 hours to warm up the cache:

python3 manage.py warmup-exercise-api-cache --force

This way it will reset the TTL of the exercise cache every 24 hours. This work fine and it is blazing fast now ;-) I think updating a exercise will warmup it's cache induvidually so, i don't have to wait till the next day for the updates (but i am not sure).

Proposal

It would be nice if the warmup-exercise-api-cache could be refactured similar to sync_exercises, so that celery task can be created. This would eliminate the fact that a cronjob is needed to warmup the cache.

[rolandgeider]

mhh, yes, that does sound like a good idea

[wbonicki]

Can I be assigned to the task?

[rolandgeider]

Sure @wbonicki !

We already have some periodic tasks, e.g. in wger.exercises.tasks.setup_periodic_tasks, we'd do something like that. Also, you will need to start a celery task as well btw: https://wger.readthedocs.io/en/latest/development/celery.html

[wbonicki]

Here is PR: https://github.com/wger-project/wger/pull/1977

I'm unsure about one thing: I created a new task for caching API exercises, and if it's activated, it will be executed once a day (at random time). Is that correct? Since the default value of EXERCISE_CACHE_TTL is 3600 seconds (1 hour), would it make more sense to run the task every hour instead? And another thing: I'm not sure if CACHE_API_EXERCISES_CELERY_FORCE_UPDATE variable is actually necessary.

[bbkz]

Hi wbonicki,

thank you for looking into it! Yes it makes no sense to run the cache warmup once a day when it exipres a hour later. But as the cache warmup of all exercices can take quite some time, i think it is overkill to do that every hour.

I the sysadmin can configure the celery task, he can set whatever makes sense to him.

I don't have the complete overview of the caching made, but is it possible that when a edit of a exercice has been made, that it warms up the cache for this exercise?

[wbonicki]

@bbkz do you mean that during the caching process, the exercise cache is overwritten only if it has been edited?

[bbkz]

No it think i once saw in the code, when saving a exercices a method to warm up that cache was called. I was only peeking around so i am really not sure if that is the case.

The idea of my comment was: That i think maybe the whole concept on what and how long something should be cached, should be made clear.

[rolandgeider]

at the moment when an exercise is edited, its cache is deleted. That means that when calling the overview the next time, that one needs to be recreated, which should be fine.

Also, in the compose file, we set EXERCISE_CACHE_TTL to 5 hours, so once a day should be fine (this value was probably from before we did reset the cache on edit, so this could be increased, but I'd need to check that).

I would also say we don't need CACHE_API_EXERCISES_CELERY_FORCE_UPDATE, if the instance admin wants to force this, there's the manual warmup-exercise-api-cache.py command