python-blosc2
python-blosc2 copied to clipboard
Feature request: `concatenate` for `NDArray` (C-API/Python) without decompression
Request native concatenate functionality for blosc2.NDArray, exposed in both the C-API and the Python wrapper.
Requirements:
-
C-API: Provide a C function to concatenate compressed
b2ndalong a specified axis. -
Python API: Wrap the C function, mimicking the
numpy.concatenatesignature (sequence of arrays,axisparameter). Ref: NumPy Docs - Core Constraint: Must operate directly on compressed data, avoiding full decompression/recompression.
-
Assumptions: Input arrays share identical compression settings (
codec,clevel,cparams, etc.).
Use Case:
Efficiently join large, pre-compressed datasets in both low-level C applications and Python without the performance penalty of decompression/recompression. Exposing it in the C-API is key for broader integration.
blosc2.SChunk.insert_chunk is very fast. Can this feature use this, along with metadata updates?
Thanks for considering this.