xskillscore icon indicating copy to clipboard operation
xskillscore copied to clipboard

dask apply_gufunc in apply_ufunc

Open ahuang11 opened this issue 5 years ago • 4 comments

New xarray ver supports this keyword.

I think allow_rechunk might be good or expose this option to the call e.g. xr.rmse(dask_gufunc_kwargs={...}) http://xarray.pydata.org/en/stable/generated/xarray.apply_ufunc.html

dask_gufunc_kwargs (dict, optional) – Optional keyword arguments passed to dask.array.apply_gufunc if dask=’parallelized’. Possible keywords are output_sizes, allow_rechunk and meta.

ahuang11 avatar Oct 01 '20 18:10 ahuang11

This is the issue:

ValueError: dimension lat on 0th function argument to apply_ufunc with dask='parallelized' consists of multiple chunks, but is also a core dimension. To fix, either rechunk into a single dask array chunk along this dimension, i.e., ``.chunk(lat: -1)``, or pass ``allow_rechunk=True`` in ``dask_gufunc_kwargs`` but beware that this may significantly increase memory usage.

ahuang11 avatar Oct 15 '20 19:10 ahuang11

Correct me if i'm wrong but is this issue same as https://github.com/xarray-contrib/xskillscore/issues/99 ?

raybellwaves avatar Oct 15 '20 20:10 raybellwaves

same issue.

Again. I would suggest that we have xskillscore global options that can be accessed by yaml default or context manager as in xr.set_options. args to set this way. example not proposal>

  • dask: parallelized
  • allow_rechunk : False
  • more?

aaronspring avatar Oct 16 '20 16:10 aaronspring

Yeah that would be great, same for climpred. Just need to figure out how xarray does it elegantly.

bradyrx avatar Oct 16 '20 18:10 bradyrx