dask-cuda icon indicating copy to clipboard operation
dask-cuda copied to clipboard

Replace cuDF (de)serializer with cuDF spill-aware (de)serializer

Open pentschev opened this issue 6 months ago • 2 comments

Replace cuDF (de)serializer with cuDF spill-aware (de)serializer, using both together should be avoided as that will cause excessive spilling.

Additionally add:

  • Missing test of cuDF internal spill mechanism with LocalCUDACluster;
  • dask cuda worker warning to alert the user that cuDF spilling mechanism requires client/scheduler to enable it as well.

Closes #1363 .

pentschev avatar Jul 26 '24 11:07 pentschev