jesque icon indicating copy to clipboard operation
jesque copied to clipboard

Job moved from inflight queue to comma-joined queue on worker shutdown in RESET_TO_HIGHEST_PRIORITY strategy.

Open pavan-kalyan opened this issue 3 years ago • 0 comments

I have noticed that in certain scenarios (immediate worker shutdown), jobs are removed from the inflight queue and then moved to another queue.

Consider the case where I have the following queues: q1, q2, q3, q4. My workers are meant to process jobs from these queues. WorkerPoolImpl::getNextQueue() returns a string of "q1,q2,q3,q4" (for RESET_TO_HIGHEST_PRIORITY) and this is stored in curQueue variable.

During pop, the lua script (fromMultiplePriorityQueues.sha) pops from a single queue and returns the job. But when the worker is shutting down, the jobs are removed from the inflight queue and pushed to a queue "q1,q2,q3,q4". resulting in a new key on redis "namespace:queue:q1,q2,q3,q4". This code is in WorkerPoolImpl::removeInFlight.

What is the purpose of storing the jobs in this comma-joined queue? From what I could see, no worker would poll from this queue as the lua script would always pop from the individual queues.

pavan-kalyan avatar Feb 18 '22 01:02 pavan-kalyan