uv
uv copied to clipboard
astral-sh
Python dev headers missing in venv
I think python dev headers are now missing when uv installs venv; repro (start from ubuntu 22.04):
confirmed on uv==0.5.0 and 0.5.4, not tested on other versions
export $VIRTUAL_ENV=/tmp/.venv
curl -LsSf https://astral.sh/uv/0.5.0/install.sh | sh
. $HOME/.local/bin/env
uv venv --python 3.10 $VIRTUAL_ENV
. $VIRTUAL_ENV/bin/activate
uv pip install ihm
output:
[stderr]
src/ihm_format_wrap_1.7.c:203:11: fatal error: Python.h: No such file or directory
203 | # include <Python.h>
| ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
Caused by: This error likely indicates that you need to install a library that provides "Python.h" for [email protected]
Annoyingly, it used to work - I was able to install this package, but a couple of weeks ago binaries shipped with uv were swapped for some other builds, this affected even previous versions of uv.
Similar to https://github.com/astral-sh/uv/issues/8966#issuecomment-2466513707
TL;DR Try sysconfigpatcher to adjust the uv python downloads to have the proper pkgconfig and sysconfig metadata for your system.
but a couple of weeks ago binaries shipped with uv were swapped for some other builds, this affected even previous versions of uv.
This doesn't quite make sense, we didn't change previous versions of uv and the Python distributions we use are hard-coded for each release and cannot change.
If you're using a system interpreter, you should install the headers, e.g., the following works:
docker run --platform linux/x86_64 -it ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install python3-dev && uv venv && uv pip install ihm"
I can't reproduce this failure using a managed interpreter:
docker run --platform linux/x86_64 -it ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang && uv venv -p 3.10 && uv pip install ihm"
this works:
docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.10 && uv pip install ihm"
this doesn't
docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.11 && uv pip install ihm"
I am also having this issue, and using Docker is not a feasible workaround. Could UV not be modified so that installs the Python headers when creating a venv?
👋 @wjakob, the docker examples here are to provide an MRE, you can apply the same steps outside of docker.
This is on a machine where I don't have root access, hence installing the headers via apt is not an option (also, that would be for the OS Python version, which is often quite outdated). With uv not installing them as part of the virtual environment, it is impossible to build C extensions within the environment.
@arogozhnikov your failing example uses the system interpreter (3.11 is installed in the image already)
root@c0d1cea75e52:/# uv python list
cpython-3.13.0+freethreaded-linux-x86_64-gnu <download available>
cpython-3.12.7-linux-x86_64-gnu <download available>
cpython-3.11.10-linux-x86_64-gnu <download available>
cpython-3.11.2-linux-x86_64-gnu /usr/bin/python3.11
cpython-3.11.2-linux-x86_64-gnu /usr/bin/python3 -> python3.11
cpython-3.11.2-linux-x86_64-gnu /bin/python3.11
cpython-3.11.2-linux-x86_64-gnu /bin/python3 -> python3.11
cpython-3.10.15-linux-x86_64-gnu <download available>
cpython-3.9.20-linux-x86_64-gnu <download available>
cpython-3.8.20-linux-x86_64-gnu <download available>
cpython-3.7.9-linux-x86_64-gnu <download available>
pypy-3.10.14-linux-x86_64-gnu <download available>
pypy-3.9.19-linux-x86_64-gnu <download available>
pypy-3.8.16-linux-x86_64-gnu <download available>
pypy-3.7.13-linux-x86_64-gnu <download available>
That's why you need to install development headers via the system package manager.
If you force uv to use a managed interpreter with --python-preference only-manage, the example works
docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.11 --python-preference only-managed && uv pip install ihm"
docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.11 --python-preference only-managed && uv pip install ihm"
Thanks @zanieb, I've been incorrectly under assumption OP wanted to use gcc and not clang, hence my suggestions to use sysconfigpatcher.
docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "uv venv -p 3.11.10 && uv tool install git+https://github.com/bluss/sysconfigpatcher && uv tool run sysconfigpatcher /root/.local/share/uv/python/cpython-3.11.10-linux-x86_64-gnu && uv pip install ihm"
Do you think it's worth expanding in docs when is it reasonable to attempt to use sysconfigpatcher by adding a note to python-versions.md or maybe a troubleshooting faq.
Do you think it's worth expanding in docs when is it reasonable to attempt to use sysconfigpatcher by adding a note to python-versions.md or maybe a troubleshooting faq.
It's sort of a priority for me to understand the problems people are encountering and fix it here rather than recommend sysconfigpatcher — it's nice that it exists and I'm sure we'll learn from it but I'd prefer a first-party solution.
please share a reproducible example
Here's one:
FROM fedora:41
RUN dnf update -y && \
dnf group install -y 'development-tools' && \
dnf install -y \
gcc \
gcc-c++ \
gtk3 \
gtk3-devel \
libxcrypt-compat \
python3 \
python3-devel \
which
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
Run the resulting image then:
# just ensuring we don't use the system-managed one which is 3.13
[root@0a8793bee0ed ~]# uv init --python ">=3.10,<3.11" repro
[root@0a8793bee0ed ~]# cd repro
[root@0a8793bee0ed repro]# uv add 'wxpython>=4.0.0'
# ... takes a minute then fails with...
...
The configuration failed
(complete log in /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/build/waf/3.10/gtk3/config.log)
Then we go and look at the detailed logs
[root@0a8793bee0ed repro]# cat /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/build/waf/3.10/gtk3/config.log
# ... bunch of noise until ...
['/usr/bin/g++', '-g', '-fwrapv', '-O3', '-fPIC', '-fno-strict-aliasing', '-I/install/include/python3.10', '-DPYTHONDIR="/root/.cache/uv/builds-v0/.tmpn5kGVI/lib/python3.10/site-packages"', '-DPYTHONARCHDIR="/root/.cache/uv/builds-v0/.tmpn5kGVI/lib/python3.10/site-packages"', '-DNDEBUG', '../test.cpp', '-c', '-o/root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/build/waf/3.10/gtk3/.conf_check_ef50035bd89e2895b752adc35a4fe169/testbuild/test.cpp.1.o']
err: ../test.cpp:2:10: fatal error: Python.h: No such file or directory
2 | #include <Python.h>
| ^~~~~~~~~~
compilation terminated.
from /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src: Test does not build: Traceback (most recent call last):
File "/root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/bin/.waf3-2.1.2-5b48554f897113218781486acfe4d78e/waflib/Configure.py", line 337, in run_build
bld.compile()
File "/root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/bin/.waf3-2.1.2-5b48554f897113218781486acfe4d78e/waflib/Build.py", line 176, in compile
raise Errors.BuildError(self.producer.error)
waflib.Errors.BuildError: Build failed
-> task in 'testprog' failed with exit status 1 (run with -v to display more information)
Broken python installation? Get python-config now!
from /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src: The configuration failed
I can confirm that using sysconfigpatcher fixes this issue (the wxpython build takes a bit of an eternity, but it eventually does succeed)
[root@0a8793bee0ed repro]# uv tool install 'git+https://github.com/bluss/sysconfigpatcher'
Updated https://github.com/bluss/sysconfigpatcher (c1ebf8a)
Resolved 1 package in 0.49ms
Built sysconfigpatcher @ git+https://github.com/bluss/sysconfigpatcher@c1ebf8ab9274dcde255484d93ce0f1fd1f76a248
Prepared 1 package in 322ms
Installed 1 package in 0.99ms
+ sysconfigpatcher==0.3.1 (from git+https://github.com/bluss/sysconfigpatcher@c1ebf8ab9274dcde255484d93ce0f1fd1f76a248)
Installed 1 executable: sysconfigpatcher
[root@0a8793bee0ed repro]# realpath .venv/bin/python
/root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/bin/python3.10
[root@0a8793bee0ed repro]# sysconfigpatcher /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/bin/python3.10
INFO: Patched /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/_sysconfigdata__linux_x86_64-linux-gnu.py
INFO: Patched /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/pkgconfig/python-3.10-embed.pc
INFO: Patched /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/pkgconfig/python-3.10.pc
[root@0a8793bee0ed repro]# uv add 'wxpython>=4.0.0'
Resolved 4 packages in 36ms
Built wxpython==4.2.2
Prepared 1 package in 2m 58s
Installed 3 packages in 29ms
+ numpy==2.2.0
+ six==1.17.0
+ wxpython==4.2.2
Also worth noting that UV's manager isn't the only one that leads to these kinds of issues; back then I experienced a similar (but not the same exact) issue with pyenv+poetry, linked here https://gitlab.com/ita1024/waf/-/issues/2420
Thanks @Tristan971, I believe that one is a manifestation of https://github.com/indygreg/python-build-standalone/issues/374 which we're also tracking in https://github.com/astral-sh/uv/issues/8429
The workaround for this is:
export CPPFLAGS="-I/root/.local/share/uv/python/cpython-3.10.16-linux-aarch64-gnu/include/python3.10"
export LDFLAGS="-L/root/.local/share/uv/python/cpython-3.10.16-linux-aarch64-gnu/lib"
(with the proper path to the Python you're using)
As you said it's taking forever to build so I can't confirm at this second, but it's definitely past that early failure.
If it's any help, this would still fail
> docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:0.5.9-bookworm /bin/bash -c "uv venv -p 3.11.10 && uv pip install ihm"
Using CPython 3.11.10
Creating virtual environment at: .venv
Resolved 2 packages in 330ms
x Failed to download and build `ihm==1.8`
`-> Build backend failed to build wheel through `build_wheel` (exit status: 1)
[stdout]
running bdist_wheel
running build
running build_py
creating build/lib.linux-x86_64-cpython-311/ihm
copying ihm/__init__.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/dataset.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/representation.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/analysis.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/report.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/metadata.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/flr.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/cross_linkers.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/reader.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/restraint.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/source.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/startmodel.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/format.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/model.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/test.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/geometry.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/citations.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/format_bcif.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/protocol.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/multi_state_scheme.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/dictionary.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/location.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/dumper.py -> build/lib.linux-x86_64-cpython-311/ihm
copying ihm/reference.py -> build/lib.linux-x86_64-cpython-311/ihm
creating build/lib.linux-x86_64-cpython-311/ihm/util
copying ihm/util/__init__.py -> build/lib.linux-x86_64-cpython-311/ihm/util
copying ihm/util/make_mmcif.py -> build/lib.linux-x86_64-cpython-311/ihm/util
running build_ext
building 'ihm._format' extension
creating build/temp.linux-x86_64-cpython-311/src
clang -pthread -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3
-Wall -fPIC -fPIC -Isrc -I/root/.cache/uv/builds-v0/.tmpOq7G05/include
-I/root/.local/share/uv/python/cpython-3.11.10-linux-x86_64-gnu/include/python3.11 -c src/ihm_format.c -o
build/temp.linux-x86_64-cpython-311/src/ihm_format.o
[stderr]
error: command 'clang' failed: No such file or directory
@samypr100 You need to have a sane build setup (C++ dev environment) in your container. This is not related to the main issue, which was about the presence of Python headers.
@samypr100 You need to have a sane build setup (C++ dev environment) in your container. This is not related to the main issue, which was about the presence of Python headers.
Thanks, note this is a buildpack-deps based image so it has a sane build setup.
Yeah for me that's separate from this issue (though a good point from @samypr100).
My last example now also works as of 0.5.10 release without the need for CC=gcc 🥳
> docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:0.5.10-bookworm /bin/bash -c "uv venv -p 3.11.10 && uv pip install ihm"
Using CPython 3.11.10
Creating virtual environment at: .venv
Resolved 2 packages in 406ms
Built ihm==1.8
Prepared 2 packages in 2.72s
Installed 2 packages in 2ms
+ ihm==1.8
+ msgpack==1.1.0
Hello, I'm trying to understand in which way this issue is solved.
$ docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/sh -c "uv venv -p 3.13 && find .venv"
Status: Downloaded newer image for ghcr.io/astral-sh/uv:bookworm
Using CPython 3.13.5
Creating virtual environment at: .venv
.venv
.venv/CACHEDIR.TAG
.venv/.gitignore
.venv/bin
.venv/bin/python
.venv/bin/python3
.venv/bin/python3.13
.venv/bin/activate
.venv/bin/activate.csh
.venv/bin/activate.fish
.venv/bin/activate.nu
.venv/bin/activate.ps1
.venv/bin/activate.bat
.venv/bin/deactivate.bat
.venv/bin/pydoc.bat
.venv/bin/activate_this.py
.venv/pyvenv.cfg
.venv/lib
.venv/lib/python3.13
.venv/lib/python3.13/site-packages
.venv/lib/python3.13/site-packages/_virtualenv.py
.venv/lib/python3.13/site-packages/_virtualenv.pth
.venv/lib64
At no point the includes are in the venv, they end up in a $HOME/.local/share folder outside of my control. What must I do to get the python headers of the version installed by uv in a path accessible from within the .venv ? (or at least a path I can easily access programmatically without knowledge of the user's OS or $HOME).
The context is a C++ software that lives entirely outside of uv's venv but needs to access the Python headers of a python installed with a specific .venv
I don't think it's a part of the standard for includes to be present in the virtual environment. See https://peps.python.org/pep-0405/
The site and sysconfig standard-library modules are modified such that the standard library and header files are found relative to sys.base_prefix / sys.base_exec_prefix, while site-package directories (“purelib” and “platlib”, in sysconfig terms) are still found relative to sys.prefix / sys.exec_prefix.
Use sys.base_prefix?
I would like to be able to reach these folders without invoking python as part of my cmake scripts - for instance I could be cross-compiling with the python binary not being in the architecture of my host in the part of my build script which needs to know the location of this include folder
I would like to be able to reach these folders without invoking python as part of my cmake scripts - for instance I could be cross-compiling with the python binary not being in the architecture of my host in the part of my build script which needs to know the location of this include folder
Sadly includes are not in virtualenvs per spec like Zanie mentioned. You can use CMake Python3 module for this as suggested in https://github.com/astral-sh/uv/issues/10267#issuecomment-2569938357 and tell it use the virtual environment.