[WIP] Adjust all algorithms to work with CuPy

[pentschev]

Requirements:

• https://github.com/numpy/numpy/pull/13046
• https://github.com/cupy/cupy/pull/2079

Resolves #73.

[pentschev]

The build will fail now, since it depends on NumPy and CuPy fixes that weren't yet merged. However, I can confirm that all existing tests pass on my setup after the changes here.

[TomAugspurger]

Hmm looks like I dropped the ball on deprecating the estimators here... I don't really care where we do that. If people still think it makes sense for the estimators to live in dask-ml, and the optimizers here, then I'll update that PR.

On Fri, Mar 8, 2019 at 2:47 PM James Bourbeau [email protected] wrote:

@jrbourbeau commented on this pull request.

On a related note, the estimators in dask-glm were implemented in dask-ml (ref dask/dask-ml#94 https://github.com/dask/dask-ml/pull/94). I think the plan was to deprecate dask_glm.estimators (#66 https://github.com/dask/dask-glm/pull/66) but keep the optimizers and regularizers in dask-glm.

@TomAugspurger https://github.com/TomAugspurger is this still the case?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/dask-glm/pull/75#pullrequestreview-212472675, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQHIrzo9kNs6vPUN8xgMX2dPgSXdMz6ks5vUsxngaJpZM4bcQw0 .

[pentschev]

FWIW, I have no strong opinions there.

Sorry if this question has already been answered before or sounds a bit stupid, but wouldn't it then make sense to have Dask-GLM completely deprecated and move everything into Dask-ML? GLM being a statistical model, it doesn't fall too far off of ML, but of course, maybe syntactically it could be considered something very unrelated.

[jrbourbeau]

My personal preference would be to merge #66 and deprecate dask_glm.estimators in favor of dask_ml.linear_model. That way all the scikit-learn style estimators live in one place.

[TomAugspurger]

I updated #66 to fix the merge conflict.

On Fri, Mar 8, 2019 at 3:30 PM James Bourbeau [email protected] wrote:

My personal preference would be to merge #66 https://github.com/dask/dask-glm/pull/66 and deprecate dask_glm.estimators in favor of dask_ml.linear_model. That way all the scikit-learn style estimators live in one place.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/dask-glm/pull/75#issuecomment-471082728, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQHIviKDl3-lLKjpMbI8WnZ9DwKBDSRks5vUtZxgaJpZM4bcQw0 .

[matthieubulte]

Hi, this PR would be quite useful for me.

Are you only waiting for #66 to be finalized and merged to merge this one?

It seems that #66 is paused for now, is that something where I could help to unblock this PR?

Otherwise, anything else I could do to get this merged?

Thanks!

[pentschev]

As far as I recall, this PR should mostly work (if not entirely). It's not necessarily dependant upon #66, even though a little modification would be required once it's merged. However, the changes here are dependant upon several unreleased changes:

1. https://github.com/numpy/numpy/pull/13046 -- will be available in NumPy 1.17
2. https://github.com/cupy/cupy/pull/2171 -- will only be merged once NumPy 1.17 is released, then available on the next CuPy release after that
3. https://github.com/cupy/cupy/pull/2079 / https://github.com/cupy/cupy/pull/2219 -- available in CuPy 7.0.0b1, but requires setting CUPY_EXPERIMENTAL_SLICE_COPY=1 environment variable

That said, we are likely to wait to have all these changes released before we can merge that. I'm expecting that NumPy is released sometime this month, following that, probably a CuPy release with necessary changes is probably due in August or September.

[matthieubulte]

Great, thanks for the clarifications!