django-q2
django-q2 copied to clipboard
May redundant pickling and unpickling in `pusher()` and `worker()`
https://github.com/django-q2/django-q2/blob/1d1d0a9aa5332a68720d2423bd05c33290666cd6/django_q/pusher.py#L65
https://github.com/django-q2/django-q2/blob/1d1d0a9aa5332a68720d2423bd05c33290666cd6/django_q/worker.py#L57
The task_queue
here extends multiprocessing.queues.Queue
, which will pick the x in put()
and unpick x in get()
. This will spend seconds of time. See https://docs.python.org/3/library/multiprocessing.html#pipes-and-queues:
Note When an object is put on a queue, the object is pickled and a background thread later flushes the pickled data to an underlying pipe.
However, maybe it could be optimized. Noticed that:
- the
task
inasync_task()
will be picked viapack = SignedPackage.dumps(task)
, then be enqueued into broker, and... - the
task
inpusher()
will be unpicked viatask = SignedPackage.loads(task[1])
before picked intask_queue.put(task)
.
I take it for granted that directly put the picked task into task_queue
, and SignedPackage.loads
it in worker()
. It sounds good, but no one can guarantee that task_queue.put
will not pick the picked task again🤔
Finally, the above is just my personal opinion, there may be something wrong. Hope someone can provide feedback🥰