cubed icon indicating copy to clipboard operation
cubed copied to clipboard

Add map_overlap as a new core op

Open TomNicholas opened this issue 1 year ago • 4 comments

It would be nice to add map_overlap alongside map_blocks, blockwise, rechunk, and apply_gufunc.

It's currently not directly used within xarray (even within xarray.map_blocks, which builds a HLG), but maybe it could / should be used there (cc @dcherian)

Regardless I think it should be on the wishlist as it is used in some other packages. For example xgcm.apply_as_grid_ufunc uses a pattern where dask.array.map_overlap is called from within the function supplied to xarray.apply_ufunc(), (it wraps the actual numpy function, so the kwarg is dask='allowed'). This is a trick that allows parallelism along all dimensions (both core dims and broadcast dims) for a large class of array algorithms of interest (e.g. differential functions).

dask.map_overlap is mostly implemented using map_blocks.

TomNicholas avatar Jun 02 '23 13:06 TomNicholas