Peter Andreas Entschev
Peter Andreas Entschev
Distributed defines `memory_limit=0` as unlimited, which prevents spilling to disk, such behavior was erratic in dask-cuda but is fixed by #269 . The purpose of this issue is to raise...
After this is merged, Dask-ML must be adapted to this change as well. https://github.com/dask/dask-ml/blob/master/dask_ml/linear_model/glm.py#L187 Solves #77.
Requirements: * https://github.com/numpy/numpy/pull/13046 * https://github.com/cupy/cupy/pull/2079 Resolves #73.
After a bit of profiling, this is what I found out for Dask-GLM with Dask array: ``` 14339 0.139 0.000 0.814 0.000 /home/pentschev/.local/lib/python3.5/site-packages/dask/local.py:430(fire_task) 44898 19.945 0.000 19.945 0.000 {method 'acquire'...
To allow for easy testing of CuPy and Dask during this phase, a copy of the existing relevant Dask tests was created under https://github.com/dask/dask-glm/pull/75. This incurs in a lot of...
For the past few weeks, I've been working on issues related to [NEP-18](https://www.numpy.org/neps/nep-0018-array-function-protocol.html) support for Dask (and ultimately, Dask-GLM as well) to allow CuPy to be used as a backend....
As promised in #73, I'd like to share here some sample code I've been working on, and the results I've obtained. Please note, unfortunately this is not reproducible elsewhere _yet_,...
Adds Dask tests with UCX-Py support in preparation for UCXX support in https://github.com/rapidsai/cuml/pull/5697. This should help verifying all is working well with UCX-Py without required RAFT changes for UCXX support.
Replace cuDF (de)serializer with cuDF spill-aware (de)serializer, using both together should be avoided as that will cause excessive spilling. Additionally add: - Missing test of cuDF internal spill mechanism with...
Using both the cuDF internal spilling mechanism and `--device-memory-limit` together may lead to excessive spilling. The more appropriate solution is to disable cuDF Dask spilling when cuDF spilling (see #1362)...