cubed icon indicating copy to clipboard operation
cubed copied to clipboard

Comparison to Xarray-Beam

Open shoyer opened this issue 1 year ago • 10 comments

I would be curious how the cubed approach compares in performance to my Xarray-Beam library, beyond the superficial differences (NumPy vs Xarray data): https://github.com/google/xarray-beam

One issue that comes to mind with storing all data in Zarr is the regular chunk-size limitation. For example, can you efficiently rechunk between arrays with relatively prime chunk sizes (e.g., from 1000 to 999)? I think doing this efficiently requires irregular chunk sizes, or you end up rechunking everything to size 1, which can be super slow.

shoyer avatar Aug 31 '22 16:08 shoyer