cubed icon indicating copy to clipboard operation
cubed copied to clipboard

Use general blob stores for the intermediate store

Open tomwhite opened this issue 7 months ago • 1 comments

The idea here is to use a key-value object store for intermediate data chunks (suggested in https://github.com/cubed-dev/cubed/issues/601#issuecomment-2543100565). This would have a few advantages over using Zarr directly:

  • avoid memory copies (Zarr still has extra copies, see https://github.com/tomwhite/memray-array)
  • no need to create Zarr metadata
  • can store irregular chunks (in future)

Note that elimination of map_direct, and multi-stage rechunking (implemented as general blockwise operations, #700) were pre-requisites for this, since everything is now done at the block level, rather than arbitrary slices of arrays.

tomwhite avatar Jun 16 '25 15:06 tomwhite

It's worth doing #715 and/or #731 first as they are simpler to do and may well give similar performance and memory improvements as using key-value store (e.g. obstore) directly from Cubed.

tomwhite avatar Jun 17 '25 12:06 tomwhite