django-q2 icon indicating copy to clipboard operation
django-q2 copied to clipboard

May redundant pickling and unpickling in `pusher()` and `worker()`

Open HairlessVillager opened this issue 9 months ago • 0 comments

https://github.com/django-q2/django-q2/blob/1d1d0a9aa5332a68720d2423bd05c33290666cd6/django_q/pusher.py#L65

https://github.com/django-q2/django-q2/blob/1d1d0a9aa5332a68720d2423bd05c33290666cd6/django_q/worker.py#L57

The task_queue here extends multiprocessing.queues.Queue, which will pick the x in put() and unpick x in get(). This will spend seconds of time. See https://docs.python.org/3/library/multiprocessing.html#pipes-and-queues:

Note When an object is put on a queue, the object is pickled and a background thread later flushes the pickled data to an underlying pipe.

However, maybe it could be optimized. Noticed that:

  1. the task in async_task() will be picked via pack = SignedPackage.dumps(task), then be enqueued into broker, and...
  2. the task in pusher() will be unpicked via task = SignedPackage.loads(task[1]) before picked in task_queue.put(task).

I take it for granted that directly put the picked task into task_queue, and SignedPackage.loads it in worker(). It sounds good, but no one can guarantee that task_queue.put will not pick the picked task again🤔

Finally, the above is just my personal opinion, there may be something wrong. Hope someone can provide feedback🥰

HairlessVillager avatar May 09 '24 15:05 HairlessVillager