numcodecs ENH: Free-Threading (No-GIL) Support for Python 3.13+

xref https://github.com/zarr-developers/numcodecs/issues/670

I took a quick look, and I'm not wholly sure on what parts of the locking, possibly synchronizing access to the c-bloc library. Worst case scenario, maybe each of the codecs need to be checked for support.

The standard TODOs for adding free-threading support are in two phases, the first of which is a baseline consisting of the following tasks:

[ ] Run the test suite with pytest-run-parallel to find potential issues, and fix them.
[ ] Run the test suite under ThreadSanitizer. If possible, depends on how many dependencies there are and if they run under TSan.
[ ] Add cp313t-* to CI to build free-threading wheels.

The next phase is a bit more invasive, and it involves a single task:

[ ] Audit Python bindings and declare them free-threading compatible (xref https://py-free-threading.github.io/porting/#updating-extension-modules).

For more details, please see the suggested plan of attack in the py-free-threading guide. The two phase approach is discussed in https://github.com/explosion/cymem/issues/47#issuecomment-2894478856 and https://github.com/gorakhargosh/watchdog/issues/1105#issuecomment-2888439578.

Note that this is the first time I've looked at this repo, so I might be missing known issues or code that needs closer inspection. Any suggestions here will be very useful. I'll start with the build without warning PR as also discussed in #670.

Let me know if I ought to hold off for any reason. (including if @tomwhite is working on it).

May 26 '25 03:05 HaoZeke

One of the issues is zfpy

DEBUG No compatible version found for: numcodecs[zfpy]
  × No solution found when resolving dependencies:
  ╰─▶ Because only the following versions of zfpy are available:
          zfpy<=1.0.0
          zfpy==1.0.0.post1
          zfpy==1.0.0.post3
          zfpy==1.0.1
      and zfpy==1.0.0 has no wheels with a matching Python implementation tag (e.g., `cp313`), we can conclude that zfpy>=1.0.0,<1.0.0.post1 cannot be used.
      And because zfpy>=1.0.0.post1,<=1.0.0.post3 was yanked (reason: arm64 wheels still broken), we can conclude that zfpy>=1.0.0.post1,<=1.0.0.post3 cannot
      be used.
      And because zfpy==1.0.1 has no wheels with a matching Python ABI tag (e.g., `cp313t`) and numcodecs[zfpy]==0.16.2.dev0+dirty depends on zfpy>=1.0.0, we
      can conclude that numcodecs[zfpy]==0.16.2.dev0+dirty cannot be used.
      And because only numcodecs[zfpy]==0.16.2.dev0+dirty is available and you require numcodecs[zfpy], we can conclude that your requirements are
      unsatisfiable.

      hint: You require CPython 3.13 (`cp313t`), but we only found wheels for `zfpy` (v1.0.1) with the following Python ABI tags: `cp39`, `cp310`, `cp311`,
      `cp312`, `cp313`

      hint: You require CPython 3.13 (`cp313`), but we only found wheels for `zfpy` (v1.0.0) with the following Python implementation tags: `cp37`, `cp38`,
      `cp39`, `cp310`, `cp311`, `cp312`

May 26 '25 03:05 HaoZeke

pcodec also seems to cause immediate segfaults...

https://github.com/HaoZeke/numcodecs/actions/runs/15246098627/job/42873119696

Test run

============================= test session starts ==============================
platform darwin -- Python 3.13.3, pytest-8.3.5, pluggy-1.6.0 -- /Users/runner/work/numcodecs/numcodecs/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /Users/runner/work/numcodecs/numcodecs
configfile: pyproject.toml
testpaths: numcodecs/tests
plugins: timeout-2.4.0, run-parallel-0.4.2, cov-6.1.1
timeout: 600.0s
timeout method: signal
timeout func_only: False
Fatal Python error: Segmentation fault

Current thread 0x00000001e8d30f80 (most recent call first):
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1320 in create_module
  File "<frozen importlib._bootstrap>", line 813 in module_from_spec
  File "<frozen importlib._bootstrap>", line 921 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pcodec/__init__.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1026 in exec_module
collecting ... 
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "/Users/runner/work/numcodecs/numcodecs/numcodecs/pcodec.py", line 5 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1026 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "/Users/runner/work/numcodecs/numcodecs/numcodecs/__init__.py", line 144 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1026 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1387 in _gcd_import
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1310 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1387 in _gcd_import
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1310 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1387 in _gcd_import
  File "/Users/runner/.local/share/uv/python/cpython-3.13.3+freethreaded-macos-aarch64-none/lib/python3.13t/importlib/__init__.py", line 88 in import_module
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/pathlib.py", line 587 in import_path
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 493 in importtestmodule
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 546 in _getobj
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 284 in obj
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 562 in _register_setup_module_fixture
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 549 in collect
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 389 in collect
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 341 in from_call
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 391 in pytest_make_collect_report
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 567 in collect_one_node
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 835 in _collect_one_node
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 970 in genitems
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 975 in genitems
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 809 in perform_collect
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 347 in pytest_collection
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 336 in _main
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pytest/__main__.py", line 9 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, numcodecs.compat_ext, numcodecs.blosc, numcodecs.zstd, numcodecs.lz4, numcodecs._shuffle, numcodecs.jenkins, crc32c._crc32c, numcodecs.vlen, numcodecs.fletcher32 (total: [11](https://github.com/HaoZeke/numcodecs/actions/runs/15246098627/job/42873119699#step:12:12))
Error: Process completed with exit code 139.

May 26 '25 04:05 HaoZeke

pcodec is an optional dependency, but the current blocker is blosc.

https://github.com/HaoZeke/numcodecs/actions/runs/15246191211/job/42873362395

Failure Log

numcodecs/__init__.py:43: in <module>
    from numcodecs import blosc
E   RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'numcodecs.blosc', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
________________ ERROR collecting numcodecs/tests/test_zarr3.py ________________
numcodecs/tests/test_zarr3.py:14: in <module>
    zarr = pytest.importorskip("zarr")
.venv/lib/python3.13t/site-packages/zarr/__init__.py:2: in <module>
    from zarr.api.synchronous import (
.venv/lib/python3.13t/site-packages/zarr/api/synchronous.py:7: in <module>
    import zarr.api.asynchronous as async_api
.venv/lib/python3.13t/site-packages/zarr/api/asynchronous.py:12: in <module>
    from zarr.core.array import (
.venv/lib/python3.13t/site-packages/zarr/core/__init__.py:9: in <module>
    from zarr.core.codec_pipeline import BatchedCodecPipeline  # noqa: F401
.venv/lib/python3.13t/site-packages/zarr/core/codec_pipeline.py:20: in <module>
    from zarr.core.metadata.v2 import _default_fill_value
.venv/lib/python3.13t/site-packages/zarr/core/metadata/__init__.py:3: in <module>
    from .v2 import ArrayV2Metadata, ArrayV2MetadataDict
.venv/lib/python3.13t/site-packages/zarr/core/metadata/v2.py:47: in <module>
    CompressorLikev2: TypeAlias = dict[str, JSON] | numcodecs.abc.Codec | None
E   AttributeError: module 'numcodecs' has no attribute 'abc'
************************** pytest-run-parallel report **************************
All tests were run in parallel! 🎉
=========================== short test summary info ============================
ERROR numcodecs/tests/test_astype.py - RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'numcodecs.blosc', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
ERROR numcodecs/tests/test_zarr3.py - AttributeError: module 'numcodecs' has no attribute 'abc'
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!

May 26 '25 04:05 HaoZeke

Thanks for looking at this @HaoZeke! I'm not working on this, so I'm happy for you to continue.

I think while investigating it's OK to force the GIL to be diabled. This would allow testing with Blosc.

Building free-threaded wheels would be good to get working too, as it would allow people to experiment with free-threaded Python on Zarr and the wider stack. Now that Cython 3.1 is out, it should be straightforward to change the Numcodecs wheels build to include cp313t.

May 27 '25 15:05 tomwhite