distributed icon indicating copy to clipboard operation
distributed copied to clipboard

rootish tasks with resources restrictions are not queued

Open vianneylanguille opened this issue 1 month ago • 0 comments

Hi.

I am using client.submit with resources restrictions for root tasks (only CPU only, not GPU) Then these root tasks results (futures) are set as input to subsequent client.submit calls (which are then non root tasks) that need to run on GPU only.

In the scheduler.py, the function is_rootish returns False whenever a task has resources, or worker restrictions.

I don't understand why. This cause queuing to be INOP in my case, and memory gradually increasing up to cluster saturation because root tasks (CPU bound tasks) are fulfilled faster than the actual results consumption (non root GPU bound), basically because I have much more CPU workers that GPU workers.

Is this intended, and if yes why ? Thanks

vianneylanguille avatar Dec 12 '25 20:12 vianneylanguille