django-background-tasks
django-background-tasks copied to clipboard
DBTaskRunner.get_task_to_run might cause OOM if large queue
Greetings,
I found that in class DBTaskRunner
, in method get_task_to_run
, there's a filtering of task_name
done with in-memory list. It might cause Out-Of-Memory issue when the task queue is extremely large.
https://github.com/arteria/django-background-tasks/blob/c97f959538dd1fbb678b2e2daceb3873ca16a7ab/background_task/tasks.py#L244-L246
I figure out a work around of this issue. By replace the filtering with Django's QuerySet filter
function like this:
available_tasks = Task.objects.find_available(queue) \
.filter(task_name__in=tasks._tasks.keys())[:5]
The snippet works good for my project. And hope this does help someone who has the same issue.
Thanks for highlighting and sharing this @vagrants0140 . Maybe we will pick this up and integrate in a future release.