dask-cuda icon indicating copy to clipboard operation
dask-cuda copied to clipboard

New version bug

Open raphael7777777 opened this issue 1 year ago • 3 comments

I had a code which ran a delayed function as a task. I would run this task with different inputs various times. For this reason i included gc.collects inside the function. When running, i could see in the dask dashboard that every task was taking progressively longer for no apparent reason. When i returned to the old version 23.8, everything worked normally. Even without the cg.collects inside the code, the running tasks showed some tasks (probably i assume when gc.collect ran by itself automatically) tasks would take a lot longer. Even thkugh most were taking the expected time, every 20 tasks there was one much longer, and also progressively longer

raphael7777777 avatar Nov 26 '23 17:11 raphael7777777

Could you please share more details on the problem and your environment? Ideally a minimal reproducer, combined with the output of nvidia-smi and conda list (if using conda) or pip list (if using pip)?

pentschev avatar Nov 27 '23 08:11 pentschev

Working on colab with a GPU A100. The problem i have already described above, unfortunately i cant show any code because its a proprietary algorithm. Its been solved by downgrading back to old version.

raphael7777777 avatar Nov 27 '23 11:11 raphael7777777

Unfortunately that description is insufficient to provide any help. We don't need any proprietary pieces of your code, but if you would like us to be able to look at your problem we're going to need more than that, a minimal reproducer without any of your proprietary pieces would suffice.

pentschev avatar Nov 27 '23 12:11 pentschev