Handing job already added worker to another worker

[rclai]

Is it possible to do this?

Say I have two workers that do the same type of job, and both workers get jobs. If one of the workers finishes all their jobs first and the other one still has other jobs, am I able to grab a job from that other worker?

[vsivsi]

Hi, if the worker has already grabbed the job into it's internal "queue" then the answer is no, because the JobCollection will show that job as running, even though within the worker's queue it may not actually be being worked on yet.

However, the situation you describe can only really happen if you set your worker queues to "prefetch" jobs. So this is a tradeoff. Prefetching hides the network latency for short duration jobs by requesting jobs in the background before the currently running job(s) have finished. The drawback being that if the jobs are longer running, then prefetched jobs are not available to be reassigned to other workers that may become free.

TL;DR : Only use "prefetch" for short duration jobs where throughput may be limited by the latency of the network. If your jobs run for significantly longer than the time it takes to fetch a new job, then don't use prefetch if you have multiple workers.