python-blosc2
python-blosc2 copied to clipboard
Optimise fancy indexing further
Continue to fine tune functionality and performance of fancy indexing according to outline suggested in #437
- [ ] work around for ndindex not supporting array indices separated by slices e.g. arr[0, :, [0, 1]]
- [ ] pure boolean indexing can definitely be optimised to work more like LazyExpr where for more general shapes (not just key.shape = array.shape)
- [ ] implement vindex (for numpy and blosc array return types)
- [ ] finish implementation of oindex (return blosc array)
- [ ] optimise fancy indexing for broadcasted index arrays
- [ ] implement (optimised) setitem for fancy indexing
- [ ] implement fancy indexing in slice (i.e. return blosc array not numpy array)
- [x] ~~optimise fancy indexing to avoid recalculating _slice/chunk_slice intersections every loop~~ (PR #459)
For implementing fancy indexing in slice (i.e. return blosc array not numpy array), one should return compressed output of get_fselection if the result can fit (uncompressed) in memory. If not, will have to write new code which loops over chunks of output, filling them in (possibly use chunk cache to speed things up a bit).