cubed

cubed copied to clipboard

Warn user when executing plan memory exceeds allowed memory.

1

Fixes #110. I don't know if this is a good place to include this warning (during a call to `compute()`). Since array ops are lazily evaluated, there may not be...

alxmrs

Implementation of `std` and `var`.

4

Fixes #29. Here, I use existing cubed operations to implement `var` and `std`. Please let me know if I should reimplement the primitives as pure reductions.

alxmrs

This would 1. allow the use of different storage backends in addition to the regular Zarr library - in particular #187. 2. convert the backend array API being used (NumPy,...

tomwhite

Use google tensorstore to read/write to zarr?

2

I expect you're already aware of this @tomwhite , but I wanted to ask whether or not you thought the [google-tensorstore project](https://ai.googleblog.com/2022/09/tensorstore-for-high-performance.html) might be useful in cubed. @rabernat [suggested](https://discourse.pangeo.io/t/google-tensorstore-3d-data-package/2778) benchmarking...

TomNicholas

Add map_overlap as a new core op

4

It would be nice to add `map_overlap` alongside `map_blocks`, `blockwise`, `rechunk`, and `apply_gufunc`. It's currently not directly used within xarray (even within `xarray.map_blocks`, which builds a HLG), but maybe it...

TomNicholas

Useful functions not in the Array API Standard

8

There are a few numpy functions which xarray calls on wrapped arrays but which are not (yet) in the Array API Standard. (See https://github.com/data-apis/array-api/issues/187#issuecomment-1553615779) Cubed could choose to implement these...

TomNicholas

Computations requiring irregularly-chunked Zarr stores

2

All intermediate results in Cubed are written out to persistent storage via Zarr, but currently Zarr can't represent any chunked array, because the Zarr spec does not yet support irregular...

TomNicholas

Delayed-like functionality

1

There are some cases where it would be useful to have a way of setting up a parallel map over inputs, then calling `compute` on it later - like Dask...

tomwhite

Implement something like `raise_if_cubed_computes` similar to xarray's [`raise_if_dask_computes`](https://github.com/pydata/xarray/blob/8c21376aa94deb91cb10e78f56ecd68aa7fe90fa/xarray/tests/__init__.py#L125-L130)

tomwhite

Change Coiled executor to use Dask async API

1

It would be nice if we could change `CoiledFunctionsDagExecutor` to be similar to (or based on) `AsyncDaskDistributedExecutor`, as it would then inherit all the features of that latter, like retries,...

tomwhite