uv
uv copied to clipboard
astral-sh
uv pip list / freeze / show --system do not show all system packages on Fedora
Executing uv pip list --system does not list any packages. Similarly for freeze and show subcommands (I did not test any others).
$ uv pip list --system -vv
uv_interpreter::find_python::find_default_python
0.000148s 0ms DEBUG uv_interpreter::find_python Starting interpreter discovery for default Python
0.000319s 0ms DEBUG uv_interpreter::interpreter Cached interpreter info for Python 3.12.2, skipping probing: /usr/bin/python3
0.000334s DEBUG uv::commands::pip_list Using Python 3.12.2 environment at /usr/bin/python3
OS: Fedora 39, python 3.12, amd64 uv version: 0.1.21, installed via pipx
Can you just confirm for me what packages should appear, and where they're located?
❯ pip show pip
Name: pip
Version: 23.2.1
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: [email protected]
License: MIT
Location: /usr/lib/python3.12/site-packages
$ pip list
❯ pip list
Package Version
------------------ ---------
argcomplete 2.0.0
Beaker 1.12.1
beautifulsoup4 4.12.3
blivet 3.8.2
blivet-gui 2.5.0
blurb 1.1.0
Brlapi 0.8.5
cffi 1.15.1
cfgv 3.4.0
charset-normalizer 3.2.0
click 8.1.3
colorama 0.4.6
ConfigArgParse 1.7
configobj 5.0.8
cryptography 41.0.7
cupshelpers 1.0
dasbus 1.7
dbus-python 1.3.2
distlib 0.3.7
distro 1.8.0
dnf 4.19.0
fedora-third-party 0.10
file-magic 0.4.0
filelock 3.13.1
fros 1.1
humanize 3.13.1
identify 2.5.33
idna 3.4
iniconfig 2.0.0
langtable 0.0.65
lexid 2021.1006
libcomps 0.1.20
libdnf 0.73.0
looseversion 1.3.0
lxml 4.9.3
Mako 1.2.3
MarkupSafe 2.1.3
nftables 0.1
nodeenv 1.8.0
olefile 0.46
packaging 23.1
Paste 3.5.3
pexpect 4.8.0
pid 2.2.3
pillow 10.2.0
pip 23.2.1
pipx 1.3.3
platformdirs 4.1.0
pluggy 1.3.0
ply 3.11
productmd 1.38
psutil 5.9.5
ptyprocess 0.7.0
pwquality 1.4.5
pycairo 1.25.1
pycparser 2.20
pycrypto 2.6.1
pycups 2.0.1
pycurl 7.45.2
PyGObject 3.46.0
pykickstart 3.48
pyOpenSSL 23.2.0
pyparted 3.13.0
PySocks 1.7.1
python-augeas 1.1.0
python-dateutil 2.8.2
python-meh 0.51
pyudev 0.24.1
PyYAML 6.0.1
requests 2.28.2
requests-file 1.5.1
requests-ftp 0.3.1
rpm 4.19.1.1
ruff 0.2.2
selinux 3.5
sepolicy 3.5
setools 4.4.3
setuptools 67.7.2
simpleline 1.9.0
six 1.16.0
sos 4.6.0
soupsieve 2.5
systemd-python 235
Tempita 0.5.2
terminator 2.1.3
tuna 0.5.11
urllib3 1.26.18
userpath 1.9.1
virtualenv 20.25.0
(Note that pip list also list user packages in ~/.local), which one can filter out
$ pip list --user
Package Version
-------------- ---------
blurb 1.1.0
cfgv 3.4.0
colorama 0.4.6
ConfigArgParse 1.7
distlib 0.3.7
filelock 3.13.1
identify 2.5.33
iniconfig 2.0.0
lexid 2021.1006
looseversion 1.3.0
nodeenv 1.8.0
pipx 1.3.3
platformdirs 4.1.0
pluggy 1.3.0
ruff 0.2.2
tuna 0.5.11
userpath 1.9.1
virtualenv 20.25.0
i can probably look into this sometime this week (maybe weekend) unless someone else is already on it :.)
So testing with uv version < 0.1.19 actually reveals the error:
$ uv pip list -v --system
uv_interpreter::python_query::find_default_python
0.003337s 0ms DEBUG uv_interpreter::python_query Starting interpreter discovery for default Python
0.003421s 0ms DEBUG uv_interpreter::interpreter Cached interpreter info for Python 3.12.2, skipping probing: /usr/bin/python3
0.003433s DEBUG uv::commands::pip_list Using Python 3.12.2 environment at /usr/bin/python3
error: failed to read directory `/usr/local/lib/python3.12/site-packages`
Caused by: No such file or directory (os error 2)
Directory /usr/local/lib/python3.12/site-packages indeed does not exist, uv should be looking into /usr/lib/python3.12/site-packages. LMK if I can provide more info to debug this.
So I believe this should be labeled as "bug" not "compatibility".
I believe the error is hidden in 0.1.19 above due to #2413
The error appears all the way in version 0.1.12 which introduced the system flag so this does not appear to be a regression in the original implementation.
If you actually install something with uv, it is listed when you run uv pip list?
My guess is that /usr/local/lib/python3.12/site-packages is the correct location in which to install packages, and that's where we look. But we don't look at all the entries in sys.path, which may include /usr/lib/python3.12/site-packages.
(To summarize, I think the fix here is to iterate over sys.paths (probably) instead of just looking at purelib and platlib.)
Indeed, installing something into system packages will install it to /usr/local/lib
$ sudo uv pip install tuna --system
Resolved 1 package in 261ms
Downloaded 1 package in 99ms
Installed 1 package in 3ms
+ tuna==0.5.11
$ uv pip list
Package Version
------- -------
tuna 0.5.11
$ uv pip show tuna
Name: tuna
Version: 0.5.11
Location: /usr/local/lib/python3.12/site-packages
Requires:
Here's how the site directories are configured in my Python (Fedora 39)
$ python -c "import site;print(site.getsitepackages())"
['/usr/local/lib64/python3.12/site-packages', '/usr/local/lib/python3.12/site-packages', '/usr/lib64/python3.12/site-packages', '/usr/lib/python3.12/site-packages']
(To summarize, I think the fix here is to iterate over sys.paths (probably) instead of just looking at purelib and platlib.)
Yep, that seems to be the case.
I've just checked that installing a package that already exist in /usr/lib will install it to /usr/local/lib/ but keeps the original there, so it's possible for a single package to be in multiple places and have multiple versions. So while iterating uv needs to take into account the precedence rules correctly (I guess whatever is first in the getsitepackages list gets to win?
Nope, pip will (correctly imo) only show that which will be actually used when imported in the interpreter. e.g. Things installed in /usr/local/lib take precedence before /usr/lib. (I verified this).
Note that the same logic applies to stuff installed in ~/.local. Those take precedence in the interpreter, and pip will show them instead of system packages.
I don't really agree with that decision, but good to know, thank you!
confirmed making the change below
pub fn site_packages(&self) -> impl Iterator<Item = &Path> {
let mut paths = vec![
self.interpreter.purelib(),
self.interpreter.platlib(),
Path::new(self.root()),
// Path::new("blah")
];
paths.dedup();
paths.into_iter()
}
(replace blah with other paths in sys.path, which can be built with interpreter.prefix()and others) fixes the issue, but having some trouble with path adjoining + borrowing, will continue to look.
We may need to return all of sys.path from get_interpreter_info to make this work as expected?
We may need to return all of sys.path from get_interpreter_info to make this work as expected?
oh dang, that's a very helpful pointer (i should've looked where these interpreter-specific values were coming from :.)), yah that would do it, let me try that out.
after spending a little too much effort and time i've noticed that this may have already been fixed by some other change...? (requests, which is installed by pip for system python seems to get picked up by uv via --system flag)
$ docker run -v $(pwd)/test.sh:/test.sh --entrypoint=bash fedora:39 test.sh
Package Version Location Installer
-------- -------- ----------------------------------------- ---------
dnf 4.18.2 /usr/lib/python3.12/site-packages
libcomps 0.1.20 /usr/lib64/python3.12/site-packages
libdnf 0.72.0 /usr/lib64/python3.12/site-packages
pip 23.2.1 /usr/local/lib/python3.12/site-packages pip
rpm 4.19.1.1 /usr/lib64/python3.12/site-packages
uv 0.1.26 /usr/local/lib64/python3.12/site-packages pip
[notice] A new release of pip is available: 23.2.1 -> 24.0
[notice] To update, run: python3 -m pip install --upgrade pip
uv 0.1.26
Package Version
------- -------
pip 23.2.1
uv 0.1.26
Package Version
------------------ --------
certifi 2024.2.2
charset-normalizer 3.3.2
idna 3.6
pip 23.2.1
requests 2.31.0
urllib3 2.2.1
uv 0.1.26
where
$ cat test.sh
python3 -m ensurepip &> /dev/null
pip3 install uv &> /dev/null
pip3 list --verbose
uv --version
uv pip list --system
pip3 install requests &> /dev/null
uv pip list --system
@danielhollas can you take a look to see if i'm delirious or this has indeed been fixed 🦀
the change i've been working on which uses sys.path (thanks again charlie for that pointer!) does pick up a bit more but may be somewhat unnecessary since all packages installed by pip seem to already be picked up without it (below is the snippet with the sys.path change, where uv picks up dnf and libdnf, which are not picked up with the latest version of uv).
bash-5.2# ./uv pip list --system --verbose
DEBUG Starting interpreter discovery for default Python
DEBUG Cached interpreter info for Python 3.12.1, skipping probing: /usr/bin/python3
DEBUG Using Python 3.12.1 environment at /usr/bin/python3
DEBUG Site packages: ["/usr/local/lib/python3.12/site-packages", "/usr/local/lib64/python3.12/site-packages", "/usr/lib64/python3.12/site-packages", "/usr/lib/python3.12/site-packages"]
Package Version
------- -------
dnf 4.18.2
libdnf 0.72.0
pip 23.2.1
uv 0.1.26
bash-5.2# pip3 list --verbose
Package Version Location Installer
-------- -------- ----------------------------------------- ---------
dnf 4.18.2 /usr/lib/python3.12/site-packages
libcomps 0.1.20 /usr/lib64/python3.12/site-packages
libdnf 0.72.0 /usr/lib64/python3.12/site-packages
pip 23.2.1 /usr/local/lib/python3.12/site-packages pip
rpm 4.19.1.1 /usr/lib64/python3.12/site-packages
uv 0.1.26 /usr/local/lib64/python3.12/site-packages pip
@ChannyClaus thanks for the ping. I've tested with uv 0.1.26 and it doesn't seem to be fixed.
I am using uv installed with pipx, I wonder if that makes a difference?
hmm it still seems to pick up the packages?
$ docker run -v $(pwd)/test.sh:/test.sh --entrypoint=bash fedora:39 test.sh
Package Version Location Installer
------------ -------- --------------------------------------- ---------
argcomplete 2.0.0 /usr/lib/python3.12/site-packages
click 8.1.3 /usr/lib/python3.12/site-packages rpm
dnf 4.18.2 /usr/lib/python3.12/site-packages
libcomps 0.1.20 /usr/lib64/python3.12/site-packages
libdnf 0.72.0 /usr/lib64/python3.12/site-packages
packaging 23.1 /usr/lib/python3.12/site-packages rpm
pip 23.2.1 /usr/local/lib/python3.12/site-packages pip
pipx 1.4.3 /usr/lib/python3.12/site-packages rpm
platformdirs 3.9.1 /usr/lib/python3.12/site-packages rpm
rpm 4.19.1.1 /usr/lib64/python3.12/site-packages
userpath 1.9.2 /usr/lib/python3.12/site-packages rpm
[notice] A new release of pip is available: 23.2.1 -> 24.0
[notice] To update, run: python3 -m pip install --upgrade pip
uv 0.1.27
Package Version
------- -------
pip 23.2.1
Package Version
------------------ --------
certifi 2024.2.2
charset-normalizer 3.3.2
idna 3.6
pip 23.2.1
requests 2.31.0
urllib3 2.2.1
where
$ cat test.sh
python3 -m ensurepip &> /dev/null
yum install pipx -y &> /dev/null
pipx install uv &> /dev/null
export PATH=/root/.local/bin:$PATH
pip3 list --verbose
uv --version
uv pip list --system
pip3 install requests &> /dev/null
uv pip list --system
do you have more details around your setup? it would be ideal if you have a small reproducible example end-to-end
Hmm, strange. I am running directly on Fedora 39, I am wondering if there are some differences between Fedora 39 Docker image and normal install. Also in your case you're running under root, not sure if that makes a difference. I'll try to dig around a bit more.
I failed to make progress here due to lack of any useful logging from uv on this problem. My plan was to dive into the code and add some logging myself, but I could not get passed the cargo build due to some missing dependencies (apparently, building one of the openssl crates requires perl, which is not installed in Fedora by default. I could install it if it didn't bring with it a gazilion of other dependencies :sob: ).
tl;dr It would be helpful to me if somebody could add logging to print which paths uv is looking at when running uv pip list / show/ freeze. This would be very useful for other commands as well, see also related #2155
I failed to make progress here due to lack of any useful logging from
uvon this problem. My plan was to dive into the code and add some logging myself, but I could not get passed the cargo build due to some missing dependencies (apparently, building one of the openssl crates requires perl, which is not installed in Fedora by default. I could install it if it didn't bring with it a gazilion of other dependencies 😭 ).tl;dr It would be helpful to me if somebody could add logging to print which paths uv is looking at when running
uv pip list / show/ freeze. This would be very useful for other commands as well, see also related #2155
maybe the snippet from description in https://github.com/astral-sh/uv/pull/2636#issue-2204011586 would help? (more specifically RUN yum install pip python cmake perl rust cargo -y)
seems like the issue linked may not be completely green-lit but if/once it's green-lit i could maybe take on it?
Note for future self: Turns out one can install only specific perl modules, and not the whole perl metapackage. I ended up doing:
sudo dnf install perl-File-Copy perl-File-Compare perl-IPC-Cmd perl-FindBin
@danielhollas can you take a look to see if i'm delirious or this has indeed been fixed 🦀
@ChannyClaus to clarify, this has always worked. :-) In other words, this issue is about uv ignoring Fedora system packages that come with the Fedora installation, which are in a different directory from system packages installed by the user. (I am assuming they are doing this to prevent users from messing up the system python install).
the change i've been working on which uses sys.path (thanks again charlie for that pointer!) does pick up a bit more but may be somewhat unnecessary since all packages installed by pip seem to already be picked up without it (below is the snippet with the sys.path change, where uv picks up dnf and libdnf, which are not picked up with the latest version of uv).
I'd still argue that uv should list all the packages that are available to the Python interpreter.
I noticed the following help text for the uv venv --system-site-packages
--system-site-package
Give the virtual environment access to the system site packages directory.
Unlike `pip`, when a virtual environment is created with `--system-site-packages`,
`uv` will _not_ take system site packages into account when running commands
like `uv pip list` or `uv pip install`.
The `--system-site-packages` flag will provide the virtual environment
with access to the system site packages directory at runtime,
but it will not affect the behavior of `uv` commands.
This is very much the same issue as here, but from the text it is not clear whether this was a conscious design decision or just stating the current limitation. If the former I'd argue to reconsider. As an example, in HPC context, admins may provide a base Python environment with basic scientific packages (numpy et al) compiled specifically for the machine to get the best performance. In this case, it would be quite wrong for uv to ignore these packages and reinstall them inside the venv.
I just ran into a variant of this problem when working with layered environments that use sitecustomize.py to incorporate additional shared folders: when uv is installing into the upper environments, it fails to detect the packages that are already installed in the lower layers. (My use case is along the same lines as the HPC use case above: shared layers with big dependencies preinstalled, then smaller application environments built on top of that)
I initially thought it was working correctly because uv pip install is so much faster than pip install, but the problem became clear when archiving the upper layers slowed down dramatically (unexpectedly including additional copies of torch and nvidia-cuda in an archive is... not great).
Between *.pth files and sitecustomize.py, the only reliable way to find out what packages a Python environment can currently see installed is to ask it for its sys.path value and scan every entry in order, it simply isn't correct to assume that the only directories that matter are the ones where installation tools are told to put new components.
Running python -m site is a useful way of dumping the relevant sys.path info when investigating problems along these lines (although python -c "import sys; print(sys.path)" is better for programmatic use, as was done in #3500).
Using sitecustomize.py files to reproduce the problem should be much easier to manage in test cases than relying specifically on Linux distro Python builds that add extra directories to sys.path.
I don't advise trying to use *.pth files (even in test cases), since they have problems dealing with non-ASCII paths in Python versions prior to 3.13.
Edit: for anyone curious, the specific environment layering project referred to here is https://pypi.org/project/venvstacks/
I just commented on #4466 as well, but @danielhollas and @ncoghlan's use cases resonate with me as well -- layered (and in my case, redistributable) virtual environments containing heavyweight packages such as torch and tensorflow.
Considering that #3500 is merged, is it fair to assume that a solution to this would come at essentially zero performance cost, since sys.path is already being evaluated as part of get_interpreter_info?
I just commented on #4466 as well, but @danielhollas and @ncoghlan's use cases resonate with me as well -- layered (and in my case, redistributable) virtual environments containing heavyweight packages such as torch and tensorflow.
Considering that #3500 is merged, is it fair to assume that a solution to this would come at essentially zero performance cost, since
sys.pathis already being evaluated as part ofget_interpreter_info?
i think you're right about the performance bit, but the tricky part (at least based on the last time i gave this a try) is to make this change without breaking the existing tests... it's possible some of the breaking tests need to be changed but it's quite risky to make changes to tests especially when it's not obviously correct to do so; additionally, this particular issue seemed to be less pressing / critical compared to some other reported issues.
With regards to the test suite -- could this be as simple as changing the test fixture(s) to create the venvs with system_site_packages = false?
(otherwise I definitely see how this can make tests brittle -- if not even non-portable -- assuming that the contents of the interpreter-wide site-packages is globally consistent is probably unrealistic to begin with?)
assuming that the contents of the interpreter-wide site-packages is globally consistent is probably unrealistic...
Yeah we can't do this.. we'd need to have a fake system interpreter available in the test context. We'd probably only have a small subset of test cases include handling packages in the "system" interpreter — the rest would be fully isolated. This is more likely to cause problems in the test suite when run by downstream packagers rather than here where we have full control over the CI environment.
For the test suite, I'd recommend testing the sitecustomize.py case rather than the system-site-packages case:
- Create a pair of regular (isolated) virtual environments (called "A" and "B" below)
- In environment B, inject a
sitecustomize.pyfile intosite-packagesthat usessite.addsitedirto integrate thesite-packagesfolder from environment A into environment B - Write the test cases to manipulate environment B ensuring that:
list/show/freezecan see the packages in both environments- if an installation request or dependency is already satisfied in environment A, it doesn't get installed into B
- if a package is upgraded in environment B, it is installed into B without attempting to uninstall it from A
- attempting to remove a package that is only present in A via environment B fails (<-- I haven't actually checked how
pip uninstallhandles this case, but refusing to uninstall seems preferable no matter whatpipdoes)
If that works as described, a venv with system-site-packages enabled should also work (without the hassle of trying to make tests that enable system-site-packages portable across different Python installations).
I hit this today as I wanted to prepare a demo using uv for an upcoming talk.
My use case is using a ubuntu:jammy / python 3.10 based docker image having heavy dependencies pre-installed in the system python.
I was hoping to create a --system-site-packages venv and start from there but it does not work as uv does not detect system packages from that venv (whereas pip does).
/app $ .venv/bin/python
Python 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/app/.venv/lib/python3.10/site-packages', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.10/dist-packages']
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/lib/python3.10', 'platstdlib': '/app/.venv/lib/python3.10', 'purelib': '/app/.venv/lib/python3.10/site-packages', 'platlib': '/app/.venv/lib/python3.10/site-packages', 'include': '/usr/include/python3.10', 'platinclude': '/usr/include/python3.10', 'scripts': '/app/.venv/bin', 'data': '/app/.venv'}
>>>
/app $ uv pip list
Package Version Editable project location
--------- ------- -------------------------
myproject 1.0 /app
pip 24.2
/app $ .venv/bin/python -m pip list
... long list of packages from /usr/lib/python3/dist-packages
