which best practices are already handled for you by CUDAWorker?

[ntabris]

There's a new doc about best practices. That's great!

I think the suggestions about RMM pool and UCX are already configured for you if you use CUDAWorker class. Is that true? Could that be said explicitly in the doc? (Or if it's not true, could it be said what of the "best practices" are and aren't done for you if you use CUDAWorker?)

[pentschev]

I think the suggestions about RMM pool and UCX are already configured for you if you use CUDAWorker class. Is that true?

No, neither is currently default. RMM must be explicitly set based no available memory and expected use case, for example, starting with a large pool may interfere with existing uses with libraries such as PyTorch (which will allocate memory outside of the RMM pool). UCX is not default because it is only available in conda-forge and we support PyPI for Dask-CUDA as well, plus it's not widely tested with interconnects other than CUDA IPC (NVLink/NVSwitch) and InfiniBand.

Could that be said explicitly in the doc? (Or if it's not true, could it be said what of the "best practices" are and aren't done for you if you use CUDAWorker?)

I guess that would be ok, would you care to open a PR comprising those changes?

[github-actions[bot]]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[ntabris]

No, neither is currently default.

Oh, that's good to know. I didn't realize this. Which is why it would be awesome if this was said in the docs somewhere :-)

I'm not in a position to open such a PR since I don't have a clear understanding of what dask-cuda does and doesn't automatically configure for you. I want to be a consumer of this document.