actions
Remove leftover `pip-23.2.dist-info` in `site-packages`
Description
While running a brand-new image, there are two .dist-info dirs for pip:
$ ls -l /opt/hostedtoolcache/Python/3.12.1/x64/lib/python3.12/site-packages
drwxrwxrwx+ 5 runner runneradmin 4096 Dec 17 22:19 pip
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.2.1.dist-info
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.3.2.dist-info
The first (pip-23.2.1.dist-info) belongs to the pip version that was initially installed, the second (pip-23.3.2.dist-info) to the updated version. Apparently this is because pip is updated using --ignore-installed, which leaves behind the old pip-23.2.1.dist-info.
Even though the latest version of pip is installed and used, the presence of the two dirs can create issues. For example, tools like safety detect the old version and report it since it has vulnerabilities, causing CI failures:
- https://github.com/python/cherry-picker/pull/102
Unless there is a valid reason to keep the old .dist-info around, I suggest removing the --ignore-installed flag, so that pip-23.2.1.dist-info is automatically removed during the pip update.
Click to see the full analysis of the issue
This initially came up because of a CI failure triggered by safety which detected an old version of pip, even though we were running the latest version. This lead to this issue:
- https://github.com/python/cherry-picker/pull/102
To debug the issue, I created the following test PR:
- https://github.com/ezio-melotti/cherry-picker/pull/1
The output shows that the latest version of pip was installed from the beginning and correctly used by the other commands, but an ls shows 2 .dist-info dirs for pip.
To double-check, I created an empty workflow that only executes the ls, and the two .dist-info are still present:
- https://github.com/ezio-melotti/cherry-picker/pull/2
I looked at the code of this repo to see how Python and pip where installed, and apparently it happens in:
https://github.com/actions/runner-images/blob/266f9413d39fc77ade974757b633ef98873c9c21/images/ubuntu/scripts/build/Install-Toolset.ps1#L50C1-L51
This loop installs all the tools, including Python, from https://github.com/actions/python-versions
The code that actually installs Python and updates pip should be:
https://github.com/actions/python-versions/blob/af22c2b8e41acf6dc7c64030339622962820df9e/installers/nix-setup-template.sh#L51-L53
Here the --ignore-installed flag is used:
-I, --ignore-installed Ignore the installed packages, overwriting them. This can break your system
if the existing package is of a different version or was installed with a
different package manager!
I'm not sure if/why this is needed, but I verified locally that this flag leaves around the old .dist-info. When --ignore-installed is not used, only a .dist-info dir is present after the upgrade:
$ python3 -m venv venv && source venv/bin/activate
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.2.dist-info
$ pip install --upgrade pip
...
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.3.2.dist-info
$ deactivate && rm -rf venv
When --ignore-installed is used, the old pip-23.2.dist-info dir is left behind after the upgrade:
$ python3 -m venv venv && source venv/bin/activate
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.2.dist-info
$ pip install --ignore-installed --upgrade pip
...
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.2.dist-info
pip-23.3.2.dist-info
Removing --ignore-installed from nix-setup-template.sh should therefore fix the issue, assuming it is not needed for other reasons.
Also note that the same flag is also used elsewhere, e.g. in install-pypy.sh.
If my analysis is correct, I can prepare a PR (or more) to remove the --ignore-installed flag.
Platforms affected
- [ ] Azure DevOps
- [X] GitHub Actions - Standard Runners
- [ ] GitHub Actions - Larger Runners
Runner images affected
- [ ] Ubuntu 20.04
- [X] Ubuntu 22.04
- [ ] macOS 11
- [ ] macOS 12
- [ ] macOS 13
- [ ] macOS 13 Arm64
- [ ] Windows Server 2019
- [ ] Windows Server 2022
Image version and build link
This was tested on the following image:
- Image: ubuntu-22.04
- Version: 20231217.2.0
It likely affects other (all?) images.
See e.g. https://github.com/ezio-melotti/cherry-picker/actions/runs/7386618932/job/20093604056?pr=2
Is it regression?
No
Expected behavior
There should be only one version of pip installed, and only one .dist-info dir that matches the installed version.
Actual behavior
There are two .dist-info dirs.
Repro steps
Run this workflow to check:
name: Check installed pip versions
on: [pull_request, push, workflow_dispatch]
jobs:
check_pip:
runs-on: ubuntu-latest
steps:
- run: ls -l /opt/hostedtoolcache/Python/3.12.1/x64/lib/python3.12/site-packages
It will output this:
drwxrwxrwx+ 5 runner runneradmin 4096 Dec 17 22:19 pip
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.2.1.dist-info
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.3.2.dist-info
Transferring to setup-python as we do not maintain the hostedtoolcache archives, we only provide them as they are in the image
Hello, @ezio-melotti ! Thank you for reporting this issue, we will look into it :)
This bug causes problems with uv, because it expects (as python does too) that there is only one version per package (https://github.com/astral-sh/uv/issues/1848#issuecomment-1959061012)
#713 is a duplicate of this
It looks like @mayeut created and merged a PR that superseded the one I made, and this should have fixed the issue:
- https://github.com/actions/python-versions/pull/268
- https://github.com/actions/python-versions/pull/254
There are a few releases in actions/python-versions dated after the merge so, unless there is something else missing, I believe this issue can be closed.
