LxL
@Digenis How do you think about this?
But the original code also iterates the queues, and `if msg is not None:` can handle concurrency problem, so can the new code. https://github.com/scrapy/scrapyd/blob/4520c45acc115d0fc766eae30788468a77690433/scrapyd/poller.py#L20-L26
poll() ends when returnValue() is called. Can you explain ‘race conditions‘ in details, maybe with an example?
How about the third commit? I save the highest priority of each queue in the queue_priority_map, and get it updated whenever needed, so that poll() can figure out the highest...
> A multiple queue consumer scenario. > Suppose a slow consumer iterates all the queues So, if *cancel.json* is called while poll() is iterating all the queues with `yield maybeDeferred`,...
> Also, I just realized that the FIFO principal is still violated. > > A limit of 2 parallel processes. > 2 empty fresh queues. > > On hour 13:00...
Even though `for p, q in iteritems(self.queues):` would violate FIFO when polling multiple projects, we can still use the priority parameter to adjust the polling order as long as we...
Changes in the fifth commit: 1. **A new column named 'insert_time' is add** when creating tables in JsonSqlitePriorityQueue: `insert_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP` 2. The table name in SqliteSpiderQueue **now defaults...
> Even if it's backwards compatible with user code, >it's not with user data in the queue. But it only ignores the pending jobs on the first startup, which may...