cubed
cubed copied to clipboard
Comparison to Xarray-Beam
I would be curious how the cubed approach compares in performance to my Xarray-Beam library, beyond the superficial differences (NumPy vs Xarray data): https://github.com/google/xarray-beam
One issue that comes to mind with storing all data in Zarr is the regular chunk-size limitation. For example, can you efficiently rechunk between arrays with relatively prime chunk sizes (e.g., from 1000 to 999)? I think doing this efficiently requires irregular chunk sizes, or you end up rechunking everything to size 1, which can be super slow.