distributed
distributed copied to clipboard
rootish tasks with resources restrictions are not queued
Hi.
I am using client.submit with resources restrictions for root tasks (only CPU only, not GPU) Then these root tasks results (futures) are set as input to subsequent client.submit calls (which are then non root tasks) that need to run on GPU only.
In the scheduler.py, the function is_rootish returns False whenever a task has resources, or worker restrictions.
I don't understand why. This cause queuing to be INOP in my case, and memory gradually increasing up to cluster saturation because root tasks (CPU bound tasks) are fulfilled faster than the actual results consumption (non root GPU bound), basically because I have much more CPU workers that GPU workers.
Is this intended, and if yes why ? Thanks