yaml
Unable to install C extension in 3.13t free threaded
Python 3.13.3 free threaded version fails to compile the C extensions for pyyaml.
Installing with:
# pip install -v --no-build-isolation pyyaml
You get the following error:
BUILDING CYTHON EXT; self.include_dirs=['/usr/local/include/python3.13t'] self.library_dirs=['/usr/local/lib'] self.define=None
building 'yaml._yaml' extension
creating build/temp.linux-x86_64-cpython-313t/yaml
gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/usr/local/include/python3.13t -c yaml/_yaml.c -o build/temp.linux-x86_64-cpython-313t/yaml/_yaml.o
yaml/_yaml.c:2155:80: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
2155 | static CYTHON_INLINE PyObject *__Pyx_PyVectorcall_FastCallDict(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw);
| ^~~~~~~~~~~~~~~~~~~~
| vectorcallfunc
yaml/_yaml.c:29665:69: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
29665 | static PyObject *__Pyx_PyVectorcall_FastCallDict_kw(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw)
| ^~~~~~~~~~~~~~~~~~~~
| vectorcallfunc
yaml/_yaml.c:29710:80: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
29710 | static CYTHON_INLINE PyObject *__Pyx_PyVectorcall_FastCallDict(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw)
| ^~~~~~~~~~~~~~~~~~~~
| vectorcallfunc
yaml/_yaml.c: In function ‘__Pyx_CyFunction_CallAsMethod’:
yaml/_yaml.c:30399:6: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
30399 | __pyx_vectorcallfunc vc = __Pyx_CyFunction_func_vectorcall(cyfunc);
| ^~~~~~~~~~~~~~~~~~~~
| vectorcallfunc
yaml/_yaml.c:2245:45: warning: initialization of ‘int’ from ‘vectorcallfunc’ {aka ‘struct _object * (*)(struct _object *, struct _object * const*, long unsigned int, struct _object *)’} makes integer from pointer without a cast [-Wint-conversion]
2245 | #define __Pyx_CyFunction_func_vectorcall(f) (((PyCFunctionObject*)f)->vectorcall)
| ^
yaml/_yaml.c:30399:32: note: in expansion of macro ‘__Pyx_CyFunction_func_vectorcall’
30399 | __pyx_vectorcallfunc vc = __Pyx_CyFunction_func_vectorcall(cyfunc);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
yaml/_yaml.c:30402:16: warning: implicit declaration of function ‘__Pyx_PyVectorcall_FastCallDict’; did you mean ‘__Pyx_PyObject_FastCallDict’? [-Wimplicit-function-declaration]
30402 | return __Pyx_PyVectorcall_FastCallDict(func, vc, &PyTuple_GET_ITEM(args, 0), (size_t)PyTuple_GET_SIZE(args), kw);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| __Pyx_PyObject_FastCallDict
yaml/_yaml.c:30402:16: warning: returning ‘int’ from a function with return type ‘PyObject *’ {aka ‘struct _object *’} makes pointer from integer without a cast [-Wint-conversion]
30402 | return __Pyx_PyVectorcall_FastCallDict(func, vc, &PyTuple_GET_ITEM(args, 0), (size_t)PyTuple_GET_SIZE(args), kw);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Error compiling module, falling back to pure Python
Installing the binary version does not include them either:
# pip install -v --no-build-isolation pyyaml --only-binary=wheel
>>> from pyyaml import CSafeLoader
Traceback (most recent call last):
File "<python-input-0>", line 1, in <module>
from pyyaml import CSafeLoader
ModuleNotFoundError: No module named 'pyyaml'
>>> from yaml import CSafeLoader
Traceback (most recent call last):
File "<python-input-1>", line 1, in <module>
from yaml import CSafeLoader
ImportError: cannot import name 'CSafeLoader' from 'yaml' (/usr/local/lib/python3.13t/site-packages/yaml/__init__.py). Did you mean: 'SafeLoader'?
Have the following versions installed:
root@ebc84fd57817:/# pip freeze | egrep -ie 'cython|setuptools'
Cython==3.0.12
setuptools==78.1.0
root@ebc84fd57817:/# pip --version
pip 25.0.1 from /usr/local/lib/python3.13t/site-packages/pip (python 3.13)
root@ebc84fd57817:/# python3 --version
Python 3.13.3
root@ebc84fd57817:/# python
Python 3.13.3 experimental free-threading build (main, Apr 15 2025, 18:16:12) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
root@ebc84fd57817:/# apt list | grep libyaml
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
libyaml-0-2/now 0.2.5-1 amd64 [installed,local]
libyaml-dev/now 0.2.5-1 amd64 [installed,local]
Using 3.13 standard, this works perfectly from a binary wheel and also compiling from source.
root@3cab962f6856:/# python3
Python 3.13.3 (main, Apr 9 2025, 00:27:54) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from yaml import CSafeLoader
>>> CSafeLoader
<class 'yaml.cyaml.CSafeLoader'>
If it helps and to give you the exact same environment, free threaded Python can be build into a Docker container will the following:
Starting with the upstream official image library here, I just add --diable-gil on line 50 and build with docker build -t python:3.13t .
#
# NOTE: THIS DOCKERFILE IS GENERATED VIA "apply-templates.sh"
#
# PLEASE DO NOT EDIT IT DIRECTLY.
#
FROM buildpack-deps:bookworm
# ensure local python is preferred over distribution python
ENV PATH /usr/local/bin:$PATH
# runtime dependencies
RUN set -eux; \
apt-get update; \
apt-get install -y --no-install-recommends \
libbluetooth-dev \
tk-dev \
uuid-dev \
; \
rm -rf /var/lib/apt/lists/*
ENV GPG_KEY 7169605F62C751356D054A26A821E680E5FA6305
ENV PYTHON_VERSION 3.13.3
ENV PYTHON_SHA256 40f868bcbdeb8149a3149580bb9bfd407b3321cd48f0be631af955ac92c0e041
RUN set -eux; \
\
wget -O python.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz"; \
echo "$PYTHON_SHA256 *python.tar.xz" | sha256sum -c -; \
wget -O python.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc"; \
GNUPGHOME="$(mktemp -d)"; export GNUPGHOME; \
gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$GPG_KEY"; \
gpg --batch --verify python.tar.xz.asc python.tar.xz; \
gpgconf --kill all; \
rm -rf "$GNUPGHOME" python.tar.xz.asc; \
mkdir -p /usr/src/python; \
tar --extract --directory /usr/src/python --strip-components=1 --file python.tar.xz; \
rm python.tar.xz; \
\
cd /usr/src/python; \
gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)"; \
./configure \
--build="$gnuArch" \
--enable-loadable-sqlite-extensions \
--enable-optimizations \
--enable-option-checking=fatal \
--enable-shared \
--with-lto \
--with-ensurepip \
--disable-gil \
; \
nproc="$(nproc)"; \
EXTRA_CFLAGS="$(dpkg-buildflags --get CFLAGS)"; \
LDFLAGS="$(dpkg-buildflags --get LDFLAGS)"; \
arch="$(dpkg --print-architecture)"; arch="${arch##*-}"; \
# https://docs.python.org/3.12/howto/perf_profiling.html
# https://github.com/docker-library/python/pull/1000#issuecomment-2597021615
case "$arch" in \
amd64|arm64) \
# only add "-mno-omit-leaf" on arches that support it
# https://gcc.gnu.org/onlinedocs/gcc-14.2.0/gcc/x86-Options.html#index-momit-leaf-frame-pointer-2
# https://gcc.gnu.org/onlinedocs/gcc-14.2.0/gcc/AArch64-Options.html#index-momit-leaf-frame-pointer
EXTRA_CFLAGS="${EXTRA_CFLAGS:-} -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"; \
;; \
i386) \
# don't enable frame-pointers on 32bit x86 due to performance drop.
;; \
*) \
# other arches don't support "-mno-omit-leaf"
EXTRA_CFLAGS="${EXTRA_CFLAGS:-} -fno-omit-frame-pointer"; \
;; \
esac; \
make -j "$nproc" \
"EXTRA_CFLAGS=${EXTRA_CFLAGS:-}" \
"LDFLAGS=${LDFLAGS:-}" \
; \
# https://github.com/docker-library/python/issues/784
# prevent accidental usage of a system installed libpython of the same version
rm python; \
make -j "$nproc" \
"EXTRA_CFLAGS=${EXTRA_CFLAGS:-}" \
"LDFLAGS=${LDFLAGS:--Wl},-rpath='\$\$ORIGIN/../lib'" \
python \
; \
make install; \
\
# enable GDB to load debugging data: https://github.com/docker-library/python/pull/701
bin="$(readlink -ve /usr/local/bin/python3)"; \
dir="$(dirname "$bin")"; \
mkdir -p "/usr/share/gdb/auto-load/$dir"; \
cp -vL Tools/gdb/libpython.py "/usr/share/gdb/auto-load/$bin-gdb.py"; \
\
cd /; \
rm -rf /usr/src/python; \
\
find /usr/local -depth \
\( \
\( -type d -a \( -name test -o -name tests -o -name idle_test \) \) \
-o \( -type f -a \( -name '*.pyc' -o -name '*.pyo' -o -name 'libpython*.a' \) \) \
\) -exec rm -rf '{}' + \
; \
\
ldconfig; \
\
export PYTHONDONTWRITEBYTECODE=1; \
python3 --version; \
pip3 --version
# make some useful symlinks that are expected to exist ("/usr/local/bin/python" and friends)
RUN set -eux; \
for src in idle3 pip3 pydoc3 python3 python3-config; do \
dst="$(echo "$src" | tr -d 3)"; \
[ -s "/usr/local/bin/$src" ]; \
[ ! -e "/usr/local/bin/$dst" ]; \
ln -svT "$src" "/usr/local/bin/$dst"; \
done
CMD ["python3"]
Okay, figured this out. Cython support for free threaded Python is in beta. This can be made to work with the following with Debian:
pip install 'Cython==3.1.0b1' setuptools
apt update && apt install -y libyaml-dev
pip install --verbose --no-build-isolation pyyaml
I have not been able to make this work on macOS.
However it looks like the C extension does not support free threading at this time:
root@f30d2a7b622f:/# python
Python 3.13.3 experimental free-threading build (main, Apr 15 2025, 18:16:12) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from yaml import CSafeLoader
<frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'yaml._yaml', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
>>>
root@f30d2a7b622f:/# python -Xgil=0
Python 3.13.3 experimental free-threading build (main, Apr 15 2025, 18:16:12) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from yaml import CSafeLoader
>>>
We are currently working on free-threaded support for PyYAML in our fork:
https://github.com/Quansight-Labs/pyyaml/pull/2
We plan on opening an upstream PR once we've finished internally reviewing the necessary changes, which will hopefully be soon.
There are unfortunately some pretty deep changes required because of how PyYAML stores state on classes and if you happen to (for example) define a resolver at runtime based on symbols found in a YAML file while parsing files in a thread pool, this can lead to badness. @lysnikolaou's solution is to use thread-local storage and replace the state in the class with a registry that manages the thread-local storage.
Hey!
After a lot of work from all the folks at Quansight, we’re excited to share that we’ve made good progress on addressing thread safety challenges in PyYAML and we’d like to explore how we can best collaborate with the maintainers to upstream those changes.
We’ve been developing on our fork which has been working relatively well in practice. We currently know of at least LibCST using our fork and @mgorny has indicated that more and more projects are starting to use it, making it more challenging for Python distributors to handle this. It’s certainly not ideal to have multiple forks for maintainers or users, particularly since the absence of free-threaded environment markers means people have to unconditionally switch to PyYAML-ft in Python 3.13+.
It would really be helpful if we could upstream the required changes to PyYAML and start sunsetting the fork before it becomes too widely used to deprecate. We’ve prepared a branch with the current diff to upstream PyYAML and we’d be more than happy to work with the PyYAML maintainers to bring it to a state where they’d consider merging it upstream.
In more detail, this branch does the following:
- Use thread-local storage in the
Constructor,RepresenterandResolvermixins to store the registries. - Add tests that verify that the thread-local storage works as it’s supposed to and, additionally, add a stress-test that generates and reparses a yaml file that looks like a Github actions workflow.
- Switch to Cython 3.1 that adds support for free-threading.
- Add the ability to build cp313t wheels.
Having said all of this, we want to emphasize that we don’t want to put any additional pressure to the maintainers here. We’re happy to keep maintaining the fork for as long as it’s needed. Ultimately, our main goal is supporting the community in whatever way works best for everyone.
cc @nitzmahone
Reading around a lot of conversation I missed on random issues and notes in the fork's README, it seems like folks misread my previous refusal to merge a related PR as a blanket rejection of FT support in PyYAML, which couldn't be further from the truth. The approach taken in #830 just YOLO flipped the flag to claim that PyYAML was FT-ready before Cython had taken care of anything (much less PyYAML itself) - an irresponsible claim given the state of, well, everything at the time.
I'm happy to work toward getting FT stuff ready here in pretty short order, though at a glance, I think the TLS registry approach being taken currently in the fork is largely unnecessary for real-world concerns and breaks a lot of existing code. I'm doing some local testing right now to verify that and will share my findings either way once that's completed- probably on a new tracker issue like we did for CFFI (where I'm also currently a supportive speed-bump for FT support).
@ngoldbaum @lysnikolaou @mgorny
See #870 for initial notes on how I'd like to proceed with getting FT support in upstream PyYAML. Scope is definitely open for discussion, but IMO the breaking changes introduced by the TLS registry in the fork is a case of "the cure is far worse than the disease", especially since the underlying API constraints around concurrent global state definition have existed for PyYAML's entire lifetime and nobody's complained thus far. Things like #700 attempt to address usability issues around PyYAML's global customization/config, but that doesn't need to be a blocker here since the underlying "problem" isn't new. Is there actually a real-world use-case where thread-agile definition of a type is something that someone wants to do?
Reading around a lot of conversation I missed on random issues and notes in the fork's README, it seems like folks misread my previous refusal to merge a related PR as a blanket rejection of FT support in PyYAML, which couldn't be further from the truth.
Having written most of what's on that README, I just wanna say that this was not at all my intention. That's why that specific part of the README says:
The PyYAML maintainers decided to not port PyYAML to the free-threaded build before the latter, along with Cython support for it, has been tested more extensively in real-world applications.
(emphasis not in the original text)
I'm sorry if I misunderstood your concerns there or if the text on that README reads differently than what I intended.
I'm happy to work toward getting FT stuff ready here in pretty short order, though at a glance, I think the TLS registry approach being taken currently in the fork is largely unnecessary for real-world concerns and breaks a lot of existing code. I'm doing some local testing right now to verify that and will share my findings either way once that's completed- probably on a new tracker issue like we did for CFFI (where I'm also currently a supportive speed-bump for FT support).
It's great to hear you're planning to work on free-threading support! Whenever that happens, we're okay sunsetting the fork and redirect any users (only libcst as far as we know) to upstream PyYAML again. Also, I'll reiterate here that we're still available to help/support you in any way we can. Please let us know in case there's anything we can do to make it easier for you to work on this. We're also available for a call in case you wanna discuss things in more detail and formulate a plan on working on this together.
especially since the underlying API constraints around concurrent global state definition have existed for PyYAML's entire lifetime and nobody's complained thus far.
Our experience thus far has been that a lot of threading issues existent in the ecosystem had not surface before free-threading just because threading wasn't the best concurrency model to use and people generally refrained from it and used other tools instead. As support for and addoption of free-threading increases, it's our expectation that such issues will become more relevant and, thus, people will need to think harder about them.
@lysnikolaou Thanks for the clarification- makes sense.
Just to make sure I'm not missing something: what was the sequence of events leading to the decision to change the type config (eg representer/constructor registration) to a per-thread registry in the fork? Was it "must eliminate all TSan warnings without ignores", trying to add importlib.reload() support, dynamic type definitions, something else? The builtin implementations (and most 3rd party usages) do imperative type setup in global module code right below the type definition, and Ansible's implementations layer in an __init_subclass__ - all of which still run under Python's per-module import lock in FT builds. Admittedly, I haven't tried really hard, but I have yet to come up with a real-world scenario where any additional locking is required. Altering the config on existing/builtin types, while possible, is a pretty gross abuse of the API that I wouldn't want to put any effort into aiding, and most other cases (reload, dynamic, etc) were already rife with issues endemic to the original API design (ie, not unique to FT).
Basically, though the existing API design doesn't explicitly enforce it, this stuff is intended to be immutable once a given type is created. Sans evidence of a compelling use case for mutability, it doesn't seem like something we should try to accommodate- if anything, I'd rather put effort into warning/enforcement around Loader/Dumper type config immutability.
what was the sequence of events leading to the decision to change the type config (eg representer/constructor registration) to a per-thread registry in the fork?
It was a combination of tests failing, TSAN warnings we wanted to eliminate and the realization that dynamic registration of (especially) resolvers could lead to parsing results changing in the midst of a parsing pass.
Basically, though the existing API design doesn't explicitly enforce it, this stuff is intended to be immutable once a given type is created. Sans evidence of a compelling use case for mutability, it doesn't seem like something we should try to accommodate- if anything, I'd rather put effort into warning/enforcement around Loader/Dumper type config immutability.
You're the one that knows PyYAML and how it's used more extensively. If you think that enforcing type config immutability is the way to go, then I'd be happy to work on that. More documentation around this stuff will certainly help as well.