cubed
cubed copied to clipboard
Use general blob stores for the intermediate store
The idea here is to use a key-value object store for intermediate data chunks (suggested in https://github.com/cubed-dev/cubed/issues/601#issuecomment-2543100565). This would have a few advantages over using Zarr directly:
- avoid memory copies (Zarr still has extra copies, see https://github.com/tomwhite/memray-array)
- no need to create Zarr metadata
- can store irregular chunks (in future)
Note that elimination of map_direct, and multi-stage rechunking (implemented as general blockwise operations, #700) were pre-requisites for this, since everything is now done at the block level, rather than arbitrary slices of arrays.
It's worth doing #715 and/or #731 first as they are simpler to do and may well give similar performance and memory improvements as using key-value store (e.g. obstore) directly from Cubed.