django-background-tasks icon indicating copy to clipboard operation
django-background-tasks copied to clipboard

DBTaskRunner.get_task_to_run might cause OOM if large queue

Open vagrants0140 opened this issue 5 years ago • 1 comments

Greetings,

I found that in class DBTaskRunner, in method get_task_to_run, there's a filtering of task_name done with in-memory list. It might cause Out-Of-Memory issue when the task queue is extremely large. https://github.com/arteria/django-background-tasks/blob/c97f959538dd1fbb678b2e2daceb3873ca16a7ab/background_task/tasks.py#L244-L246

I figure out a work around of this issue. By replace the filtering with Django's QuerySet filter function like this:

available_tasks = Task.objects.find_available(queue) \
                      .filter(task_name__in=tasks._tasks.keys())[:5]

The snippet works good for my project. And hope this does help someone who has the same issue.

vagrants0140 avatar Jun 05 '19 18:06 vagrants0140

Thanks for highlighting and sharing this @vagrants0140 . Maybe we will pick this up and integrate in a future release.

philippeowagner avatar Jul 01 '19 08:07 philippeowagner