python
gh-143120: pixi builds for free-threading and TSAN
Follow-up to #142469
Add pixi infrastructure for
-
free-threading: compiles with--disable-gil -
tsan-free-threading: compiles with--disable-gil --with-thread-sanitizer
Currently, free-threading seems to work well on Linux.
tsan crashes on build: [EDIT read comment below for solution]
[...]/build_tsan ./python -E -S -m sysconfig --generate [...]
│ │ -posix-vars ;\
[...]
│ │ Bus error (core dumped)
│ │ make: *** [Makefile:1162: pybuilddir.txt] Error 135
@kumaraditya303 @lucascolley any idea what's wrong here?
- Issue: gh-143120
@kumaraditya303 @lucascolley any idea what's wrong here?
no clue sorry I haven't seen this before
The tsan crash is due to the default mmap_rnd_bits on Ubuntu 24.04; it goes away after lowering it:
$ sudo sysctl vm.mmap_rnd_bits
vm.mmap_rnd_bits = 32 # too high
$ sudo sysctl vm.mmap_rnd_bits=28 # reduce it
vm.mmap_rnd_bits = 28
ubuntu-latest github actions workers seems to be unaffected.
There's a lot of copy/paste here, which can make maintenance harder. Does pixi have some concept of code reuse or parametrisation? Maybe not the best example, but something like GitHub Actions' reusable workflows?
@lucascolley is there anything we can do right now to reduce the duplication here? Or does that require fixes in pixi?
As mentioned in the README already, this is blocked on https://github.com/prefix-dev/pixi/issues/4599.
@lucascolley and I chatted a little about this on the pixi discord. It seems that the way these packages are defined, the numpy build in the CI for https://github.com/numpy/numpy/pull/30510 ends up having Python packages installed in a GIL-enabled 3.15 envrionment, while the build happens in a 3.15t environment:
numpy on 🎋 tsan-ft-pixi is 📦 v2.5.0.dev0 via 🐍
❯ ls -s pixi-packages/tsan-free-threading/.pixi/build/work/numpy-K8YFxN4GfUg-ME03ejjoGjs/host_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.15t/site-packages/
╭───┬────────────────────┬──────┬───────┬────────────────╮
│ # │ name │ type │ size │ modified │
├───┼────────────────────┼──────┼───────┼────────────────┤
│ 0 │ README.txt │ file │ 119 B │ 18 minutes ago │
│ 1 │ pip │ dir │ 288 B │ 12 minutes ago │
│ 2 │ pip-25.3.dist-info │ dir │ 288 B │ 12 minutes ago │
╰───┴────────────────────┴──────┴───────┴────────────────╯
numpy on 🎋 tsan-ft-pixi is 📦 v2.5.0.dev0 via 🐍
❯ ls -s pixi-packages/tsan-free-threading/.pixi/build/work/numpy-K8YFxN4GfUg-ME03ejjoGjs/host_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.15/site-packages/
╭────┬─────────────────────────────────────┬──────┬────────┬────────────────╮
│ # │ name │ type │ size │ modified │
├────┼─────────────────────────────────────┼──────┼────────┼────────────────┤
│ 0 │ Cython │ dir │ 736 B │ 12 minutes ago │
│ 1 │ cython-3.2.4.dist-info │ dir │ 320 B │ 12 minutes ago │
│ 2 │ cython.py │ file │ 632 B │ 21 hours ago │
│ 3 │ meson-1.10.0.dist-info │ dir │ 352 B │ 12 minutes ago │
│ 4 │ meson_python-0.18.0.dist-info │ dir │ 288 B │ 12 minutes ago │
│ 5 │ mesonbuild │ dir │ 1.5 kB │ 12 minutes ago │
│ 6 │ mesonpy │ dir │ 288 B │ 12 minutes ago │
│ 7 │ packaging │ dir │ 576 B │ 12 minutes ago │
│ 8 │ packaging-25.0.dist-info │ dir │ 352 B │ 12 minutes ago │
│ 9 │ pyproject_metadata │ dir │ 256 B │ 12 minutes ago │
│ 10 │ pyproject_metadata-0.10.0.dist-info │ dir │ 288 B │ 12 minutes ago │
│ 11 │ pyximport │ dir │ 160 B │ 12 minutes ago │
│ 12 │ tomli │ dir │ 224 B │ 12 minutes ago │
│ 13 │ tomli-2.3.0.dist-info │ dir │ 288 B │ 12 minutes ago │
╰────┴─────────────────────────────────────┴──────┴────────┴────────────────╯
This is either a bug in rattler-build or it's a bug in the package definitions here.
If it's the latter - maybe the package definitions here need to define the ABI pin that the conda-forge python-freethreading package defines? Or some other way to indicate the Python ABI in the metadata for the package. Maybe @h-vetinari or @isuruf have ideas.
The python builds here (technically added in the previous PRs) do not have the correct metadata to be used as a conda package. Either you should fix the metadata here or not use them downstream.
You need the python_abi run requirement which in turn require a proper build string.
In order to not let this repo be held up against conda-forge/python_abi, you'll need to add python_abi to the recipe as well.
@lucascolley and I chatted a little about this on the pixi discord. It seems that the way these packages are defined, the numpy build in the CI for numpy/numpy#30510 ends up having Python packages installed in a GIL-enabled 3.15 envrionment, while the build happens in a 3.15t environment:
Does this imply that if I use the ASAN numpy recipe from numpy main I end up running with a regular (non-ASAN) cpython?
However what you're saying doesn't seem to add up. If I create a project
[workspace]
channels = ["https://prefix.dev/conda-forge"]
platforms = ["linux-64", "linux-aarch64", "osx-64", "osx-arm64"]
preview = ["pixi-build"]
[dependencies]
python.git = "https://github.com/crusaderky/cpython"
python.subdirectory = "Tools/pixi-packages/tsan-free-threading"
python.rev = "tsan"
numpy.git = "https://github.com/ngoldbaum/numpy"
numpy.subdirectory = "pixi-packages/tsan-free-threading"
numpy.rev = "tsan-ft-pixi"
then I get
$ pixi ls
Name Version Build Size Kind Source
_libgcc_mutex 0.1 conda_forge 2.50 KiB conda https://prefix.dev/conda-forge
_openmp_mutex 4.5 2_gnu 23.07 KiB conda https://prefix.dev/conda-forge
bzip2 1.0.8 hda65f42_8 254.24 KiB conda https://prefix.dev/conda-forge
ca-certificates 2026.1.4 hbd8a1cb_0 143.08 KiB conda https://prefix.dev/conda-forge
icu 78.1 h33c6efd_0 12.13 MiB conda https://prefix.dev/conda-forge
libexpat 2.7.3 hecca717_0 74.85 KiB conda https://prefix.dev/conda-forge
libffi 3.5.2 h9ec8514_0 56.47 KiB conda https://prefix.dev/conda-forge
libgcc 15.2.0 he0feb66_16 1018.36 KiB conda https://prefix.dev/conda-forge
libgomp 15.2.0 he0feb66_16 589.14 KiB conda https://prefix.dev/conda-forge
liblzma 5.8.1 hb9d3cd8_2 110.25 KiB conda https://prefix.dev/conda-forge
libmpdec 4.0.0 hb9d3cd8_0 89.05 KiB conda https://prefix.dev/conda-forge
libsqlite 3.51.1 hf4e2dac_1 921.34 KiB conda https://prefix.dev/conda-forge
libstdcxx 15.2.0 h934c35e_16 5.59 MiB conda https://prefix.dev/conda-forge
libuuid 2.41.3 h5347b49_0 39.37 KiB conda https://prefix.dev/conda-forge
libxcb 1.17.0 h8a09558_0 386.61 KiB conda https://prefix.dev/conda-forge
libzlib 1.3.1 hb9d3cd8_2 59.53 KiB conda https://prefix.dev/conda-forge
ncurses 6.5 h2d0b736_3 870.74 KiB conda https://prefix.dev/conda-forge
numpy 2.5.0.dev0 hb0f4dca_0 conda git+https://github.com/ngoldbaum/numpy?subdirectory=pixi-packages%2Ftsan-free-threading&rev=tsan-ft-pixi#069e0d56b11769b8f2df85ca7a08e5c23928a75b
openssl 3.6.0 h26f9b46_0 3.02 MiB conda https://prefix.dev/conda-forge
pthread-stubs 0.4 hb9d3cd8_1002 8.06 KiB conda https://prefix.dev/conda-forge
python 3.15 hb0f4dca_0 conda git+https://github.com/crusaderky/cpython?subdirectory=Tools%2Fpixi-packages%2Ftsan-free-threading&rev=tsan#9a2a8963286077536f67b89a2123355aeb049a3d
readline 8.3 h853b02a_0 336.99 KiB conda https://prefix.dev/conda-forge
tk 8.6.13 noxft_ha0e22de_103 3.13 MiB conda https://prefix.dev/conda-forge
xorg-libx11 1.8.12 h4f16b4b_0 816.30 KiB conda https://prefix.dev/conda-forge
xorg-libxau 1.0.12 hb03c661_1 14.96 KiB conda https://prefix.dev/conda-forge
xorg-libxdmcp 1.1.5 hb03c661_1 20.11 KiB conda https://prefix.dev/conda-forge
zstd 1.5.7 hb78ec9c_6 587.28 KiB conda https://prefix.dev/conda-forge
which looks healthy? I can't see any GIL-enabled python from conda-forge?
You need the python_abi run requirement which in turn require a proper build string.
I'm trying to understand what this means by reading https://github.com/conda-forge/python-feedstock/blob/main/recipe/meta.yaml.
Are you saying that what we're misssing is something like this?
outputs:
run_exports:
noarch:
- python
weak:
- python_abi {{ ver2 }}.* *_{{ abi_tag }}
{% if py_interp_debug == "yes" %}
- python {{ ver2 }}.* *_debug_{{ abi_tag }}
{% endif %}
https://github.com/conda-forge/python-feedstock/blob/main/recipe/meta.yaml#L36-L40 looks most relevant to the failure Nathan is seeing. I think it's just the wrong directory rather than the wrong package being installed.
Actually the exact problem seems described at https://github.com/conda-forge/python-feedstock/blob/main/recipe/build_base.sh#L526-L541
So maybe we need an equivalent of https://github.com/conda/conda/issues/14053 fixed in rattler
however rattler seems to get it right when installing python-freethreading from conda-forge at least:
~/sandbox/rattler-pyt via 🧚 v0.62.2
❯ pixi add python-freethreading cowpy
✔ Added python-freethreading >=3.14.2,<4
✔ Added cowpy >=1.1.5,<2
~/sandbox/rattler-pyt via 🧚 v0.62.2 took 2s
❯ ls -s .pixi/envs/default/lib/python3.14t/site-packages
╭───┬───────────────────────┬──────┬───────┬────────────────╮
│ # │ name │ type │ size │ modified │
├───┼───────────────────────┼──────┼───────┼────────────────┤
│ 0 │ README.txt │ file │ 119 B │ a month ago │
│ 1 │ conda-site.pth │ file │ 256 B │ 22 seconds ago │
│ 2 │ cowpy │ dir │ 128 B │ 23 seconds ago │
│ 3 │ cowpy-1.1.5.dist-info │ dir │ 320 B │ 23 seconds ago │
╰───┴───────────────────────┴──────┴───────┴────────────────╯
can you try this diff Guido?
diff --git a/Tools/pixi-packages/tsan-free-threading/recipe.yaml b/Tools/pixi-packages/tsan-free-threading/recipe.yaml
index dfae06ad7a5..3417e8b92b1 100644
--- a/Tools/pixi-packages/tsan-free-threading/recipe.yaml
+++ b/Tools/pixi-packages/tsan-free-threading/recipe.yaml
@@ -18,4 +18,6 @@ build:
env:
PYTHON_VARIANT: "tsan-free-threading"
+ python:
+ site_packages_path: "lib/python3.13t/site-packages"
# derived from https://github.com/conda-forge/python-feedstock/blob/main/recipe/meta.yaml
can you try this diff Guido?
Maybe I did something wrong, but I attempted to apply this patch over at https://github.com/ngoldbaum/cpython/commit/39df6fe96a2da23b09a7f8930eff0cb1f9259862 and then updated numpy at https://github.com/ngoldbaum/numpy/commit/d28bebc4463f05045ba717ef1bc91f08a1cbf388 and it doesn't seem to work, unfortunately, at least on my local setup. CI will test it when a runner picks it up:
https://github.com/numpy/numpy/actions/runs/20754129965/job/59591290742?pr=30510
It looks like the option in the recipe isn't having any effect:
bash-5.3$ ls lib/python3.15/site-packages/
Cython meson_python-0.18.0.dist-info mesonpy pyproject_metadata tomli
cython-3.2.4.dist-info meson-1.10.0.dist-info packaging pyproject_metadata-0.10.0.dist-info tomli-2.3.0.dist-info
cython.py mesonbuild packaging-25.0.dist-info pyximport
bash-5.3$ ls lib/python3.15t/site-packages/
pip pip-25.3.dist-info README.txt
I tried it locally and it at least works with pixi global install:
Tools/pixi-packages/tsan-free-threading on 🎋 tsan [🔨?] via 🧚 v0.62.2
❯ pixi global install -e pyt --path ./python-3.15-h60d57d3_0.conda
└── pyt (installed)
├─ dependencies: python 3.15
└─ exposes: idle3, idle3.15, pip3, pip3.15, pydoc3, pydoc3.15, python, python3, python3-config, python3.15, python3.15-config, python3.15t, python3.15t-config
Tools/pixi-packages/tsan-free-threading on 🎋 tsan [🔨?] via 🧚 v0.62.2 took 2s
❯ pixi global install -e pyt cowpy
└── pyt (installed)
├─ dependencies: python 3.15, cowpy 1.1.5
└─ exposes: idle3, idle3.15, pip3, pip3.15, pydoc3, pydoc3.15, python, python3, python3-config, python3.15, python3.15-config, python3.15t, python3.15t-config, cowpy
Tools/pixi-packages/tsan-free-threading on 🎋 tsan [🔨?] via 🧚 v0.62.2
❯ ls -s ~/.pixi/envs/pyt/lib/python3.13t/site-packages/
╭───┬───────────────────────┬──────┬───────┬───────────────╮
│ # │ name │ type │ size │ modified │
├───┼───────────────────────┼──────┼───────┼───────────────┤
│ 0 │ cowpy │ dir │ 128 B │ 2 minutes ago │
│ 1 │ cowpy-1.1.5.dist-info │ dir │ 320 B │ 2 minutes ago │
╰───┴───────────────────────┴──────┴───────┴───────────────╯
ah probably that only works for noarch packages and we need something like https://github.com/conda-forge/python-feedstock/blob/c90e3cfe0a44db1b0741359d68c330cd7d861b16/recipe/sitecustomize.py
ah probably that only works for noarch packages and we need something like https://github.com/conda-forge/python-feedstock/blob/c90e3cfe0a44db1b0741359d68c330cd7d861b16/recipe/sitecustomize.py
That workaround is for older conda versions not handling noarch packages properly.
Hmm yeah on second thought meson-python should be in the right place, let me try to reproduce Nathan's failure
can you try this diff Guido?
Maybe I did something wrong, but I attempted to apply this patch over at ngoldbaum@39df6fe and then updated numpy at ngoldbaum/numpy@d28bebc and it doesn't seem to work, unfortunately, at least on my local setup. CI will test it when a runner picks it up:
numpy/numpy/actions/runs/20754129965/job/59591290742?pr=30510
doh should have checked the CI log earlier, it is working! Just worked locally for me too
doh should have checked the CI log earlier, it is working! Just worked locally for me too
Oh hey, it worked! That's so strange - I wonder what's different on my setup. It looks like pixi is up-to-date...
if in doubt try pixi clean, maybe something is being cached that shouldn't be
if in doubt try pixi clean, maybe something is being cached that shouldn't be
Nope, doesn't work. Here's the build log after pixi clean and pixi clean cache: https://gist.github.com/ngoldbaum/8ade90dd534410c6c13e349f23e2d077
The only difference I see compared with the working build on CI is that my build used uv 0.9.22 but the CI build used 0.9.21.
Nope, doesn't work. Here's the build log after
pixi cleanandpixi clean cache: https://gist.github.com/ngoldbaum/8ade90dd534410c6c13e349f23e2d077
This ended up coming down to there being a .pixi folder in the pixi-packages/tsan-free-threading subfolder of my numpy clone. pixi clean didn't remove it and it took me a while to realize it was actually in a subfolder. Sorry for all the noise on that.
looks great, https://github.com/prefix-dev/pixi-build-backends/pull/532 should get us down to a single recipe.yaml, and hopefully in a future Pixi release once we can select a variant/build string we can get down to a single pixi.toml. Seems like the latter is blocked on https://github.com/prefix-dev/pixi/issues/5248.
Are symbolic links allowed? (Especially since these files end up in the python source distribution, it might be bad because of windows). If not we might have to duplicate the files.