python-blosc2 icon indicating copy to clipboard operation
python-blosc2 copied to clipboard

Optimise fancy indexing further

Open lshaw8317 opened this issue 8 months ago • 1 comments

Continue to fine tune functionality and performance of fancy indexing according to outline suggested in #437

  • [ ] work around for ndindex not supporting array indices separated by slices e.g. arr[0, :, [0, 1]]
  • [ ] pure boolean indexing can definitely be optimised to work more like LazyExpr where for more general shapes (not just key.shape = array.shape)
  • [ ] implement vindex (for numpy and blosc array return types)
  • [ ] finish implementation of oindex (return blosc array)
  • [ ] optimise fancy indexing for broadcasted index arrays
  • [ ] implement (optimised) setitem for fancy indexing
  • [ ] implement fancy indexing in slice (i.e. return blosc array not numpy array)
  • [x] ~~optimise fancy indexing to avoid recalculating _slice/chunk_slice intersections every loop~~ (PR #459)

lshaw8317 avatar Jul 25 '25 15:07 lshaw8317

For implementing fancy indexing in slice (i.e. return blosc array not numpy array), one should return compressed output of get_fselection if the result can fit (uncompressed) in memory. If not, will have to write new code which loops over chunks of output, filling them in (possibly use chunk cache to speed things up a bit).

lshaw8317 avatar Aug 26 '25 09:08 lshaw8317