numcodecs icon indicating copy to clipboard operation
numcodecs copied to clipboard

Preliminary version of Blosc2 module

Open FrancescAlted opened this issue 2 years ago • 2 comments

This is a preliminary version of a blosc2.pyx module for numcodecs. It is essentially the same than blosc.pyx, but the API has been updated to use Blosc2 instead. A new C-Blosc2 submodule has been added too.

However, C-Blosc2 requires cmake to build, and we normally use something like scikit-build-core. Unfortunately this means quite a bit of changes into the current build procedure based on setuptools; would you be up to adopt these changes? If you don't, probably Blosc2 can be made to work with existing setuptools, but we still need to figure out how.

TODO:

  • [ ] Update setup.py / pyproject.toml to compile the C-Blosc2 library
  • [ ] Unit tests and/or doctests in docstrings
  • [ ] Tests pass locally
  • [ ] Docstrings and API docs for any new/modified user-facing classes and functions
  • [ ] Changes documented in docs/release.rst
  • [ ] Docs build locally
  • [ ] GitHub Actions CI passes
  • [ ] Test coverage to 100% (Codecov passes)

FrancescAlted avatar Sep 13 '23 13:09 FrancescAlted

I think this is an important step to take in the right direction.

Later, we should also consider a deprecation procedure for writing Blosc v1. Reading Blosc v1 should be able continue indefinitely via the c-blosc2 library

@FrancescAlted, do you have a timeline for how much longer c-blosc1 will be maintained?

mkitti avatar Sep 20 '23 14:09 mkitti

@FrancescAlted, do you have a timeline for how much longer c-blosc1 will be maintained?

Well, the Blosc team would be supporting C-Blosc1 for as long as possible. Having said that, all the action is happening in C-Blosc2 right now, and C-Blosc1 will only receive fixes in the future.

FrancescAlted avatar Oct 13 '23 10:10 FrancescAlted