django-db-queue icon indicating copy to clipboard operation
django-db-queue copied to clipboard

Add --drain flag to worker command

Open j4mie opened this issue 2 years ago • 1 comments

python manage.py worker --drain will start a worker process which will drain all jobs in the queue (if there are any) and then exit. This addresses #16 (from seven years ago 😱) and #52, and is an alternative (and much simpler) solution to that proposed in #56.

j4mie avatar Jan 21 '23 21:01 j4mie

Thanks @j4mie. I agree that it is simple and that there is attraction in that. I'd previously gone for the more complex #56 --shift_limit approach, specifically because of the potential issue @jordaneb raises. The dynamic between --rate expected volumes of jobs, average job execution time and CRON interval is a complex one. The unknown is what undesirable side effects may be introduced into the data should there be multiple workers processing the same queue. That said, the issue cannot be easily mitigated in a CRON, environment, whatever the approach, so the 'warning' may be the most pragmatic. One comment on the test for --drain consider adding an extra step to submit a further Job after the worker has initially drained the queue to check that it doesn't get processed.

collinr3 avatar Jan 22 '23 14:01 collinr3