Tom White

Fixed in #622

Thanks @TomNicholas, this is very interesting!

Thanks for doing this work @balanz24! It would be interesting to see if the results changed with larger datasets on the same quadratic means. In particular, does the optimal value...

> Particulary I've used a >300GB dataset, setting `optimize_graph=False` to avoid fusing operations in order to have stages with more than 1000 workers, as you suggested. I wouldn't set `optimize_graph=False`...

That's a great suggestion. I'll sketch out something here to start with. A Cubed executor is a subclass of `DagExecutor`: https://github.com/cubed-dev/cubed/blob/e7ff3651467f6a7c5ab2f87a211d3f4581e5f370/cubed/runtime/types.py#L9-L15 The `execute_dag` method is responsible for taking a Cubed...

From the documentation [xarray.apply_ufunc](https://docs.xarray.dev/en/stable/generated/xarray.apply_ufunc.html#xarray.apply_ufunc): > Note that due to the overhead [xarray.map_blocks()](https://docs.xarray.dev/en/stable/generated/xarray.map_blocks.html#xarray.map_blocks) is considerably slower than apply_ufunc. Also, `xarray.map_blocks` is marked as experimental and has not been implemented in [cubed-xarray](https://github.com/cubed-dev/cubed-xarray?tab=readme-ov-file#sharp-edges-),...

The interesting case is HPC, where there are multiple nodes (and hence the possibility of stragglers), but an intermediate store that uses a shared filesystem that does not support atomic...

> Using icechunk to store intermediate data might be helpful for resuming computations - each completed stage of the plan would write a new commit to one icechunk store that...

I was discussing this with @paraseba and @jhamman at the Zarr Summit last week, and they suggested that you could periodically commit work (every m minutes, or after every n...

See also https://github.com/sgkit-dev/vcztools/issues/112