dask-cuda icon indicating copy to clipboard operation
dask-cuda copied to clipboard

Switch on `Jit-Unspill` by default

Open VibhuJawa opened this issue 3 years ago • 5 comments

We should switch Jit-Unspill on by default when RMM is being used. This configuration is often over looked by users when setting up their clusters which causes them not to use memory effectively .

We should default this to True for users when they are using dask-cuda clusters with RMM POOL.

Additional Context:

A couple of cugraph+dask users recently ran into this issue and adding jit-unspill helped them relieve that pressure.

CC: @rlratzel , @jnke2016 , @quasiben , @randerzander

VibhuJawa avatar Sep 20 '22 01:09 VibhuJawa

I don't know if this is something we are ready to do yet for the general use case, JIT-Unspill for CuPy is still very slow.

cc @madsbk

pentschev avatar Sep 20 '22 06:09 pentschev

I don't know if this is something we are ready to do yet for the general use case, JIT-Unspill for CuPy is still very slow.

cc @madsbk

I thought we switched off splilling for cuPY in this PR https://github.com/rapidsai/dask-cuda/pull/568

VibhuJawa avatar Sep 20 '22 18:09 VibhuJawa

JIT-Unspill should support CuPy arrays by always un-spilling them before task execution: https://github.com/rapidsai/dask-cuda/pull/856.

madsbk avatar Sep 21 '22 17:09 madsbk

JIT-Unspill should support CuPy arrays by always un-spilling them before task execution: #856.

But last I tested it was slow, in slower than default spilling https://github.com/rapidsai/dask-cuda/issues/840#issuecomment-1026013073 . I should probably revisit that.

pentschev avatar Sep 22 '22 19:09 pentschev

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

github-actions[bot] avatar Oct 22 '22 20:10 github-actions[bot]