taskiq icon indicating copy to clipboard operation
taskiq copied to clipboard

Question about max prefetch

Open realitix opened this issue 1 year ago • 1 comments

I would like to know precisely how max_prefetch works. If I set workers = 6, for example, thus six worker processes, and I do not set max_prefetch, how will the task distribution among the 6 workers happen? Will a single worker retrieve all the tasks, or will the task distribution be balanced among the 6? For instance, if we have max_async_task set to 6, should max_prefetch also be set to 6 ?

Thanks

realitix avatar Jul 25 '24 21:07 realitix

workers are spawned as child processes max_async_task and max_prefetch are set for each worker max_prefetch means how many messages (tasks) would be in queue awaiting for execution max_async_task + max_prefetch - maximum number of tasks that are already running (max_async_task) or waiting to be executed (max_prefetch) on a single worker

u dont need to set max_prefetch to 6 it's possible that one worker would retrieve all the tasks

Sobes76rus avatar Jul 26 '24 09:07 Sobes76rus