Compile fails on 3.14 no-GIL

[clin1234]

  
  × Building wheel for zstandard (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      <string>:41: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      <string>:42: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      <frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module '_cffi_backend', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
      not modified: 'build/zstandard/_cffi.c'
      generating build/zstandard/_cffi.c
      (already up-to-date)
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-314
      creating build/lib.linux-x86_64-cpython-314/zstandard
      copying zstandard/backend_cffi.py -> build/lib.linux-x86_64-cpython-314/zstandard
      copying zstandard/__init__.py -> build/lib.linux-x86_64-cpython-314/zstandard
      copying zstandard/__init__.pyi -> build/lib.linux-x86_64-cpython-314/zstandard
      copying zstandard/py.typed -> build/lib.linux-x86_64-cpython-314/zstandard
      running build_ext
      building 'zstandard.backend_c' extension
      creating build/temp.linux-x86_64-cpython-314
      creating build/temp.linux-x86_64-cpython-314/c-ext
      x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fPIC -Ic-ext -Izstd -I/tmp/venv/include -I/usr/include/python3.14t -c c-ext/backend_c.c -o build/temp.linux-x86_64-cpython-314/c-ext/backend_c.o -DZSTD_SINGLE_FILE -DZSTDLIB_VISIBLE= -DZDICTLIB_VISIBLE= -DZSTDERRORLIB_VISIBLE= -fvisibility=hidden
      c-ext/backend_c.c: In function ‘safe_pybytes_resize’:
      c-ext/backend_c.c:316:15: error: ‘PyObject’ {aka ‘struct _object’} has no member named ‘ob_refcnt’
        316 |     if ((*obj)->ob_refcnt == 1) {
            |               ^~
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for zstandard
  Building wheel for cryptography (pyproject.toml) ... error
  error: subprocess-exited-with-error

[jimkring]

I worked around this quite a while back.

https://github.com/indygreg/python-zstandard/pull/201

I'm able to pip install git+https://github.com/jimkring/python-zstandard-nogil@676226d8adb6db384bf945ffb7ed015ba0295d4d (zstandard-0.22.0.dev0) into Python 3.13rc1 nogil (running the python3.13t.exe optional binary that the Windows Installer offers in the install options).

I tried to sync the latest indygreg/python-zstandard changes into my branch but it's not working out of the box -- it probably requires a few more tweaks to get things working again.

[ngoldbaum]

@jimkring thanks for this!

@indygreg do you think you'd be willing to publish free-threaded wheels for zstandard (assuming there are no thread-safety issues you're aware of)?

You can follow https://py-free-threading.github.io/porting/ for porting instructions.

[geertj]

Would love to see this as well. Note that a free threading build is now available in Python 3.13 which was released on October 7. Free threading is still experimental and not the default. For my specific use case it performs extremely well and so I'd like to see zstandard support this.

[dpdani]

Hi there 👋

Is there any blocker other than using Py_REFCNT? Anything I can help with?

[ngoldbaum]

Is there any blocker other than using Py_REFCNT?

I did a short grep through the codebase and don't see any obvious issues in the C extensions like mutable static global variables.

The zstd.h header talks about using one context per thread in multithreaded environments. It looks like the ZstdDecompressor extension type stores a ZSTD_DCtx on the struct for the object, so I guess that might be possibly problematic if someone shares a decompressor across threads? I didn't look at the code carefully enough to see if this would be problematic in the GIL-enabled build as well, but assuming the extension drops the GIL it probably is an issue.

Not sure if there are other thread safety issues. Running something like pytest-run-parallel on the test suite would probably catch issues.

[ofek]

@indygreg Would you be open to one of us adding support for this and you publishing a release?

[dpdani]

I did some tests with the pytest-run-parallel package and there was only one failed test, I'll post a PR when I get the chance, probably this week.

[dpdani]

Hello 👋 I've opened #243

There are a couple issues though:

• locally I see one test failing with 3.13t (and not with 3.13), I'm not sure whether it's a problem with py-zstd or with hypothesis
• I had to disable the cffi build dependency from pyproject.toml because it's not compatible with free threading
  • see https://github.com/python-cffi/cffi/issues/126
  • there is currently no way to select dependencies for free-threading/default builds in pyproject.toml
  • I guess this may be a blocker? (py-zstd can be used without cffi)

[ngoldbaum]

I'll try to take a look at that failing test this week.

There is active work going on to get cffi working - it's a headache elsewhere too (cryptography and jupyter's dependencies stand out).

[dunwoj-hlevel]

Anyone know the current status on this? I am unable to install using python3.13t

[dpdani]

The opened PR #243 is stale because of maintainer inactivity.

Currently there's work going on to port zstd into python 3.14 stdlib, if you're willing to wait for that. Alternatively, you could clone the PR code and build the package yourself, it should just work.

[dpdani]

See here for the relevant work on cpython's stdlib: https://github.com/python/cpython/issues/132983

[dunwoj-hlevel]

Thanks @dpdani - will just build it myself for now while I wait for the cpython addition.

[clin1234]

Should be resolved in these PRs: https://github.com/indygreg/python-zstandard/pull/273 https://github.com/indygreg/python-zstandard/pull/274 https://github.com/indygreg/python-zstandard/pull/275 https://github.com/indygreg/python-zstandard/pull/276

[indygreg]

The main branch now builds and passes tests on 3.13 and 3.14 free-threaded CPython. See docs/news.rst for more details.