Build fails on 3.14 no-GIL

[clin1234]

$ make
cython msgpack/_cmsgpack.pyx
python setup.py build_ext -i -f
running build_ext
building 'msgpack._cmsgpack' extension
creating build
creating build/temp.linux-x86_64-cpython-314
creating build/temp.linux-x86_64-cpython-314/msgpack
x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fPIC -I. -I/tmp/venv/include -I/usr/include/python3.14t -c msgpack/_cmsgpack.c -o build/temp.linux-x86_64-cpython-314/msgpack/_cmsgpack.o
msgpack/_cmsgpack.c:2343:80: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
 2343 | ject *__Pyx_PyVectorcall_FastCallDict(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw);
      |                                                       ^~~~~~~~~~~~~~~~~~~~
      |                                                       vectorcallfunc
msgpack/_cmsgpack.c: In function ‘__pyx_pf_7msgpack_9_cmsgpack_6Packer_2__dealloc__’:
msgpack/_cmsgpack.c:6941:3: warning: ‘Py_OptimizeFlag’ is deprecated [-Wdeprecated-declarations]
 6941 |   if (unlikely(__pyx_assertions_enabled())) {
      |   ^~
In file included from /usr/include/python3.14t/Python.h:72,
                 from msgpack/_cmsgpack.c:16:
/usr/include/python3.14t/cpython/pydebug.h:13:37: note: declared here
   13 | Py_DEPRECATED(3.12) PyAPI_DATA(int) Py_OptimizeFlag;
      |                                     ^~~~~~~~~~~~~~~
msgpack/_cmsgpack.c: At top level:
msgpack/_cmsgpack.c:21382:69: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
21382 | t *__Pyx_PyVectorcall_FastCallDict_kw(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw)
      |                                                       ^~~~~~~~~~~~~~~~~~~~
      |                                                       vectorcallfunc
msgpack/_cmsgpack.c:21427:80: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
21427 | ject *__Pyx_PyVectorcall_FastCallDict(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw)
      |                                                       ^~~~~~~~~~~~~~~~~~~~
      |                                                       vectorcallfunc
msgpack/_cmsgpack.c: In function ‘__Pyx_CyFunction_CallAsMethod’:
msgpack/_cmsgpack.c:22116:6: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
22116 |      __pyx_vectorcallfunc vc = __Pyx_CyFunction_func_vectorcall(cyfunc);
      |      ^~~~~~~~~~~~~~~~~~~~
      |      vectorcallfunc
msgpack/_cmsgpack.c:2433:45: warning: initialization of ‘int’ from ‘vectorcallfunc’ {aka ‘struct _object * (*)(struct _object *, struct _object * const*, long unsigned int,  struct _object *)’} makes integer from pointer without a cast [-Wint-conversion]
 2433 | #define __Pyx_CyFunction_func_vectorcall(f) (((PyCFunctionObject*)f)->vectorcall)
      |                                             ^
msgpack/_cmsgpack.c:22116:32: note: in expansion of macro ‘__Pyx_CyFunction_func_vectorcall’
22116 |      __pyx_vectorcallfunc vc = __Pyx_CyFunction_func_vectorcall(cyfunc);
      |                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
msgpack/_cmsgpack.c:22119:16: warning: implicit declaration of function ‘__Pyx_PyVectorcall_FastCallDict’; did you mean ‘__Pyx_PyObject_FastCallDict’? [-Wimplicit-function-declaration]
22119 |         return __Pyx_PyVectorcall_FastCallDict(func, vc, &PyTuple_GET_ITEM(args, 0), (size_t)PyTuple_GET_SIZE(args), kw);
      |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                __Pyx_PyObject_FastCallDict
msgpack/_cmsgpack.c:22119:16: warning: returning ‘int’ from a function with return type ‘PyObject *’ {aka ‘struct _object *’} makes pointer from integer without a cast [-Wint-conversion]
22119 |         return __Pyx_PyVectorcall_FastCallDict(func, vc, &PyTuple_GET_ITEM(args, 0), (size_t)PyTuple_GET_SIZE(args), kw);
      |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
make: *** [Makefile:5: all] Error 1

[methane]

no plan until cython support no-gil build.

[rgommers]

Note that while Cython support for free-threaded Python isn't fully complete yet, it works well enough in master now (3.1.0-dev, not in 3.0.x). A nightly wheel for Cython can be installed as documented on https://py-free-threading.github.io/ci/. It's used by projects with a large amount of Cython code like SciPy and scikit-learn for their free-threaded nightly wheels.

This issue should automatically go away, see https://py-free-threading.github.io/debugging/#cython-compilation-errors-unknown-type-name-__pyx_vectorcallfunc

[tacaswell]

The sdist for 1.1.0rc1 on pypi contains a couple files generated by cython which makes the install fail even if you have a version of cython that will work. Is it possible to make cython a build-time dependency and regenerate the output as needed?

[methane]

See #539.

[methane]

no-GIL Python is experimental. Do not ask support for it. Just wait until stable Cython release, or manually build instead of pip install. That is what you should do it by yourself when using experimental software.

[synodriver]

Hi. You can use this script to compile for nogil python, remember to install cython from git.

#!/usr/bin/env python
import os
import sys

from setuptools import Extension, setup
from Cython.Build import cythonize
from Cython.Compiler.Version import version as cython_version
from packaging.version import Version

PYPY = hasattr(sys, "pypy_version_info")

libraries = []
macros = []
ext_modules = []

if sys.platform == "win32":
    libraries.append("ws2_32")
    macros = [("__LITTLE_ENDIAN__", "1")]

if not PYPY and not os.environ.get("MSGPACK_PUREPYTHON"):
    ext_modules.append(
        Extension(
            "msgpack._cmsgpack",
            sources=["msgpack/_cmsgpack.pyx"],
            libraries=libraries,
            include_dirs=["."],
            define_macros=macros,
        )
    )
del libraries, macros

compiler_directives = {
    "cdivision": True,
    "embedsignature": True,
    "boundscheck": False,
    "wraparound": False,
}


if Version(cython_version) >= Version("3.1.0a0"):
    compiler_directives["freethreading_compatible"] = True

setup(
    ext_modules=cythonize(ext_modules, compiler_directives = compiler_directives),
    packages=["msgpack"],
)

I've just compile it for 3.13t.

[ngoldbaum]

Here's a patch on top of current main to get things to build: https://gist.github.com/ngoldbaum/ea251bb5abb10c1b46fc6bceca8ac606

[m-clare]

With the recent update to Cython 3.1.1, msgpack-python builds with 3.13t and passes all existing tests. However, there don't seem to be any existing tests for multithreading.

I added a multithreading test for packing to this fork, and get flakey results with segmentation fault errors:

test/test_multithreading.py::test_multithread_packing Fatal Python error: Segmentation fault

The standard TODOs for adding free-threading support are:

• [ ] Audit Python bindings and declare them free-threading compatible (xref https://py-free-threading.github.io/porting/#updating-extension-modules).
• [ ] Run the test suite with pytest-run-parallel to find potential issues, and fix them.
• [ ] Run the test suite under ThreadSanitizer. If possible, depends on how many dependencies there are and if they run under TSan.
• [ ] Add cp313t-* to CI to build free-threading wheels.

For more details, please see the suggested plan of attack in the py-free-threading guide.

Let me know if help is wanted to build out multithreading tests, like the one above, as well as to make the underlying code thread safe. Any suggestions here will be very useful.

[methane]

It is not relating to free threading. I don't support using one Packer from multiple threads, regardless free threding or with GIL.

[ngoldbaum]

I don't support using one Packer from multiple threads, regardless free threding or with GIL.

Fair enough. I don't see that documented anywhere - maybe it should be?

If you're interested in avoiding the possibility of crashing the interpreter if someone happens to try to do something unsupported like this, you can probably turn it into a runtime error.

One approach is to use an atomic integer flag that hangs off of the Packer object to indicate that it is currently in use. If someone tries to simultaneously use a Packer in two threads, the second thread sees the "in use" flag set to 1 and raises a runtime exception. You can do something similar with a threading.Lock if you use lock.acquire(blocking=False) and raise an exception if acquiring the lock fails.

[lysnikolaou]

@methane Would you be open to reviewing a PR that adds support for free-threading now that there's a stable Cython 3.1 release and PEP 779 has been accepted?

I've done a more thorough review of the code and it looks like there's no global state. The only real problem is when sharing Packer/Unpacker objects. Since you've signified that that isn't supported (and I've checked that that is the case in other language bindings for msgpack as well), I can open a PR documenting that. Optionally, we can raise an error, when detecting that an object is shared across threads.

Additionally, I'm preparing a PR to add CI and build free-threaded wheels. Is that something you would be okay with?

[methane]

Wait for two weeks. When Python 3.14RC1 is released, I start development for msgpack v1.2 in main branch. It will use new Cython and cibuildwheel.

[lysnikolaou]

@methane Python 3.14rc1 has been released. If you start working on free-threading support and need any help from my side, please let me know!

[clin1234]

I have a (draft) PR #641 that supports this, and includes a test for multi-threading.

Interesting, it only fails for no-GIL interpreters with the pure Python fallback

[lysnikolaou]

@methane Is there any way we can support you in implementing support for free-threading? The open PR is in a relatively good state, though it'll definitely need another set of eyes on it.

[ngoldbaum]

msgpack 1.1.2 has free-threaded wheels: https://pypi.org/project/msgpack/#files

I think this can be closed now.

[methane]

That wheels are not no-GIL ready. It works on free threaded Python, but it enables GIL. I decided to release one version to support Python 3.14 before supporting no-GIL.

[lysnikolaou]

@methane Maybe it's worth it to add a note on the docs? If so, I can help by creating a first draft. Before doing that, however, I'd like to understand better what you think in terms of what guarantees the library makes. I saw that on #654 you added @cython.critical_section to the public APIs for Packer and Unpacker objects. That to me means that objects can be shared, but users are responsible for ordering when using these in a multithreaded context. Is that correct?