[BUG] setuptools>=60.0.0 causes virtualenv to create abnormal tree structure on ubuntu 22.04

[danielhoherd]

setuptools version

setuptools 60, 61, 62

59.x was the last version I found to not cause trouble. This also happens to be the default version installed with ubuntu 22.04 via apt.

Python version

Python 3.10

OS

Ubuntu 22.04

Also, I was unable to reproduce this on python:3.10-buster

Additional environment information

No response

Description

Installing setuptools==62.1.0 causes virtualenv venv to create an abnormal directory structure. Uninstalling setuptools solves the problem. This is 100% reproducible in ubuntu:22.04 docker image (ubuntu@sha256:06b5d30fabc1fc574f2ecab87375692299d45f8f190d9b71f512deb494114e1f)

The failure of virtualenv also affects poetry, which has no error checking for venv activate and will attempt to modify system packages when it thinks the venv is activated but it is not.

Without setuptools, venv/bin exists. With setuptools, venv/local/bin exists and no venv/bin dir exists.

Expected behavior

Setuptools does not interfere with virtualenv directory structure.

How to Reproduce

1. Create Dockerfile:

   FROM ubuntu:22.04
   
   RUN apt-get update
   RUN apt-get install -y python3 python3-pip python3-virtualenv tree
   RUN pip install setuptools==62.1.0 ; \
       virtualenv venv ; \
       tree -L 3 venv ; \
       . venv/bin/activate
   

2. Build it.
3. Notice error.

Output

$ cat Dockerfile
FROM ubuntu:22.04

RUN apt-get update
RUN apt-get install -y python3 python3-pip python3-virtualenv tree
RUN pip install setuptools==62.1.0 ; \
    virtualenv venv ; \
    tree -L 3 venv ; \
    . venv/bin/activate
$ docker build -t test1 .
Sending build context to Docker daemon  24.01MB
Step 1/4 : FROM ubuntu:22.04
 ---> f0b07b45d05b
Step 2/4 : RUN apt-get update
 ---> Using cache
 ---> f3a881e3158b
Step 3/4 : RUN apt-get install -y python3 python3-pip python3-virtualenv tree
 ---> Using cache
 ---> 1b02afb3b2a1
Step 4/4 : RUN pip install setuptools==62.1.0 ;     virtualenv venv ;     tree -L 3 venv ;     . venv/bin/activate
 ---> Running in 5a7b54dc3a32
Collecting setuptools==62.1.0
  Downloading setuptools-62.1.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 8.8 MB/s eta 0:00:00
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.6.0
    Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed setuptools-62.1.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
created virtual environment CPython3.10.4.final.0-64 in 308ms
  creator CPython3Posix(dest=/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.0.2, setuptools==59.6.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
venv
|-- lib
|   `-- python3.10
|-- local
|   |-- bin
|   |   |-- activate
|   |   |-- activate.csh
|   |   |-- activate.fish
|   |   |-- activate.nu
|   |   |-- activate.ps1
|   |   |-- activate_this.py
|   |   |-- deactivate.nu
|   |   |-- pip
|   |   |-- pip-3.10
|   |   |-- pip3
|   |   |-- pip3.10
|   |   |-- python -> /usr/bin/python3
|   |   |-- python3 -> python
|   |   |-- python3.10 -> python
|   |   |-- wheel
|   |   |-- wheel-3.10
|   |   |-- wheel3
|   |   `-- wheel3.10
|   `-- lib
|       `-- python3.10
`-- pyvenv.cfg

6 directories, 19 files
/bin/sh: 1: .: cannot open venv/bin/activate: No such file
The command '/bin/sh -c pip install setuptools==62.1.0 ;     virtualenv venv ;     tree -L 3 venv ;     . venv/bin/activate' returned a non-zero code: 2

[danielhoherd]

Workaround / fix

I have verified with 62.1.0 that changing $HOME/.local/lib/python3.10/site-packages/distutils-precedence.pth from:

import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'local') == 'local'; enabled and __import__('_distutils_hack').add_shim();

to

import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'stdlib') == 'local'; enabled and __import__('_distutils_hack').add_shim();

Solves the problem.

See also

• https://github.com/pypa/setuptools/blob/main/CHANGES.rst#v6000
• https://github.com/pypa/setuptools/compare/v59.8.0..v60.0.0#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7L57

[abravalheri]

Thank you very much @danielhoherd for reporting this.

I did a quick experiment with Fedora:

> docker run --rm -it fedora:35

python3 -VV
# Python 3.10.2 (main, Jan 17 2022, 00:00:00) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)]

dnf install -y tree python3-pip python3-virtualenv
dnf remove -y python3-setuptools

pip --version
# pip 21.2.3 from /usr/lib/python3.10/site-packages/pip (python 3.10)
virtualenv --version
# virtualenv 20.8.1 from /usr/lib/python3.10/site-packages/virtualenv/__init__.py

python3 -m pip install setuptools==62.1.0 ; \
    virtualenv venv ; \
    tree -L 3 venv ; \
    . venv/bin/activate

And this seems to be working fine (unless there is something wrong with the script above, or something I did not manage to reproduce quite right):

venv
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate.nu
│   ├── activate.ps1
│   ├── activate_this.py
│   ├── deactivate.nu
│   ├── pip
│   ├── pip-3.10
│   ├── pip3
│   ├── pip3.10
│   ├── python -> /usr/bin/python3
│   ├── python3 -> python
│   ├── python3.10 -> python
│   ├── wheel
│   ├── wheel-3.10
│   ├── wheel3
│   └── wheel3.10
├── lib
│   └── python3.10
│       └── site-packages
├── lib64
│   └── python3.10
│       └── site-packages
└── pyvenv.cfg

If this behaviour is specific to debian based distributions... Is there any chance it is being caused by the customisations enabled by the _distutils_system_mod hook?

Regarding the suggested fix, I don't think the setuptools project wants to revert to the previous behaviour of using the local distutils...

The way best going forward would be finding out how the latest changes in https://github.com/pypa/disutils are affecting the installation schemas/templates used by virtualenv and think how we can fix that while still keeping the latest improvements.

[danielhoherd]

@abravalheri I was able to replicate your fedora experiment. You may be right that this is specific to debian distros, but it seems only new distros.

I was not able to recreate the issue on:

• ubuntu:focal
• debian:buster

But I was able to recreate my original report using the following images:

• debian:sid (debian@sha256:8bab053cb1881ffba9c0ddc2cbe6716ec65acac60084d1499eca97c00a042294)
• debian:bookworm (debian@sha256:98979ea4b0da8aecb57d7a6c69d95ada083f1b44894db7cf027e88066c14cb1c)

Here's a quick snip to test out the original condition:

IMAGE="debian@sha256:ebe4b9831fb22dfa778de4ffcb8ea0ad69b5d782d4e86cab14cc1fded5d8e761"
docker run --rm -ti "$IMAGE" /bin/bash -xc 'apt-get update && apt-get install -y python3 python3-pip tree ; hash -r ; pip3 install virtualenv setuptools==62.1.0 ; virtualenv venv ; tree -L 3 venv ; . venv/bin/activate ;'

I was unable to reproduce this on fedora 35, 36, 37

[abravalheri]

Hi @danielhoherd, I also checked it with Alpine Linux and could not reproduce the error:

> docker run --rm -it alpine:3.15.4

apk add --update python3 py3-pip tree
python3 -m pip install -U setuptools==62.1.0 virtualenv
python3 -m virtualenv venv
tree -L 3 venv

venv
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate.nu
│   ├── activate.ps1
│   ├── activate_this.py
│   ├── deactivate.nu
│   ├── pip
│   ├── pip-3.9
│   ├── pip3
│   ├── pip3.9
│   ├── python -> /usr/bin/python3
│   ├── python3 -> python
│   ├── python3.9 -> python
│   ├── wheel
│   ├── wheel-3.9
│   ├── wheel3
│   └── wheel3.9
├── lib
│   └── python3.9
│       └── site-packages
└── pyvenv.cfg

There seems to be a chance the potential fix requires some coordination between _distutils_system_mod and the usage virtualenv makes of distutils. Maybe also pypa/distutils.

@jaraco do you have any thoughts about this issue?

[nagylzs]

I just upgraded multiple computers to Ubuntu 22.04 and all of them have this problem. Replacing os.environ.get(var, 'local') with os.environ.get(var, 'stdlib') fixes the problem.

[epomatti]

Having this issue on WSL Ubuntu 22.04. This export SETUPTOOLS_USE_DISTUTILS=stdlib works around the issue.

[danielhoherd]

The following snip passes on every failing debian based image listed above:

docker run -e SETUPTOOLS_USE_DISTUTILS=stdlib --rm -ti "$IMAGE" /bin/bash -xc 'apt-get update && apt-get install -y python3 python3-pip tree ; hash -r ; pip3 install virtualenv setuptools==62.1.0 ; virtualenv venv ; tree -L 3 venv ; . venv/bin/activate ;'

[jaraco]

There seems to be a chance the potential fix requires some coordination between _distutils_system_mod and the usage virtualenv makes of distutils. Maybe also pypa/distutils.

Yes, I think you're right.

The fact that the workarounds above happen to work around the issue is because they're disabling the transitional codepath where setuptools supplies distutils.

I observe the issue does not occur when using python -m venv so the issue is unique to virtualenv.

[jaraco]

I used this dockerfile:

from ubuntu:22.04
RUN apt update
RUN apt install -y python3 python3-pip python3-virtualenv tree
RUN pip install -U setuptools
RUN virtualenv venv
CMD tree -L 3 venv

Because it allows me to see the failed expectation but also replace it and inspect the environment.

The problem can be observed by adding a breakpoint to virtualenv/discovery/py_info.py:88:

root@9b995eb67229:~# cat /usr/lib/python3/dist-packages/virtualenv/discovery/py_info.py | head -n 95 | tail -n 15
            # we cannot use distutils at all if "venv" exists, distutils don't know it
            self.distutils_install = {}
        else:
            self.sysconfig_scheme = None
            self.sysconfig_paths = {u(i): u(sysconfig.get_path(i, expand=False)) for i in sysconfig.get_path_names()}
            self.distutils_install = {u(k): u(v) for k, v in self._distutils_install().items()}

        breakpoint()

        # https://bugs.python.org/issue22199
        makefile = getattr(sysconfig, "get_makefile_filename", getattr(sysconfig, "_get_makefile_filename", None))
        self.sysconfig = {
            u(k): u(v)
            for k, v in [
                # a list of content to store from sysconfig

And inspecting the value:

root@9b995eb67229:~# python3 -m virtualenv venv
> /usr/lib/python3/dist-packages/virtualenv/discovery/py_info.py(91)__init__()
-> makefile = getattr(sysconfig, "get_makefile_filename", getattr(sysconfig, "_get_makefile_filename", None))
(Pdb) self.distutils_install
{'purelib': 'local/lib/python3.10/dist-packages', 'platlib': 'local/lib/python3.10/dist-packages', 'headers': 'usr/include/python3.10', 'scripts': 'local/bin', 'data': 'local'}

The fact that scripts is local/bin is probably what is leading virtualenv to install executables to that directory.

My guess is that something about the assumptions made between _distutils_system_mod and virtualenv are in conflict.

Probably what we need is someone from virtualenv to analyze what expectation is being missed here and determine what's the best way to meet that expectation. @gaborbernat Do you have any advice here?

I also observe that virtualenv has a codepath that bypasses distutils if a venv scheme is supplied. Perhaps the best thing would be for debian to supply that install scheme. @FFY00 Do you have a recommendation on what Debian should be doing to support a virtualenv?

@doko42, It does appear to me you authored the patch. Will you be able to help work through this issue?

[FFY00]

I also observe that virtualenv has a codepath that bypasses distutils if a venv scheme is supplied. Perhaps the best thing would be for debian to supply that install scheme. @FFY00 Do you have a recommendation on what Debian should be doing to support a virtualenv?

Yes, the easiest way to fix this would be for Debian to provide a venv scheme, as described in https://github.com/python/cpython/issues/89576. This issue exists canonically in CPython since 3.10, due to the addition of sysconfig._get_preferred_schemes, but also exists in older versions of patched Python installations if the vendors aren't careful with their patching.

cc @stefanor

[jepler]

I've filed a bug with Debian about this problem, since as far as I can tell nobody explicitly informed them that there is a problem: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1022749

[jedie]

Ran into the same bug under Linux Mint 21 with Python 3.10

I observe the issue does not occur when using python -m venv so the issue is unique to virtualenv.

This "work-a-round" works for me, e.g.:

python3 -m venv .venv
poetry install

[danielhoherd]

I saw a setuptools CVE today so I came back to revisit this, and it appears that this is no longer an issue on ubuntu 22.04 with all the latest packages. I'm not sure what solved the issue though. I just ran a several tests with setuptools==65.6.3 (which required me to install python3.10-venv, which was version 3.10.6-1~22.04.2) and was unable to reproduce the original problem.

This is also not a problem with the current ubuntu 22.04 system setuptools, which is 59.6.0-1.2 and likely never had the problem.

[saltydk]

I saw a setuptools CVE today so I came back to revisit this, and it appears that this is no longer an issue on ubuntu 22.04 with all the latest packages. I'm not sure what solved the issue though. I just ran a several tests with setuptools==65.6.3 (which required me to install python3.10-venv, which was version 3.10.6-1~22.04.2) and was unable to reproduce the original problem.

This is also not a problem with the current ubuntu 22.04 system setuptools, which is 59.6.0-1.2 and likely never had the problem.

Likely due to this https://github.com/pypa/virtualenv/pull/2415 as I can easily recreate this when using an older virtualenv version.

[khaled4vokalz]

any progress with this? I am currently trying to run pipenv sync in my project which is failing with this error as well :face_plam: AttributeError: install_layout. Did you mean: 'install_platlib'?

I tried almost everything... even doing export SETUPTOOLS_USE_DISTUTILS=stdlib as well, which is not working as well :(

I am in Ubuntu Mantic (23.10) Python 3.10.14 pip 24.0

[abravalheri]

@khaled4vokalz please check the comments above, specially https://github.com/pypa/setuptools/issues/3278#issuecomment-1118317549, https://github.com/pypa/setuptools/issues/3278#issuecomment-1289906845.

Please note that the problem might not have an official solution for the specific combination of Python/OS/pip/setuptools versions that you are using therefore you might have to reconsider those.

[stefanor]

setuptools 60 didn't ship in Ubuntu 22.04, it came later in 23.04.

The setuptools solution to this problem is the addition of the venv scheme to the python interpreter. That isn't something that is reasonable to do in a stable Ubuntu, to resolve a bug like this, that occurs only when a package is updated to a version newer than Ubuntu ships. That's outside of what's reasonable for Ubuntu support.

I suggest leaving your system setuptools as shipped by Ubuntu, and using the system virtualenv to create virtualenvs. Use the newer setuptools inside virtualenvs.

Or use venv instead of virtualenv.

[jaraco]

As I review this, I believe there's nothing more for Setuptools to do here, so I'm closing the ticket. Happy to re-open if that's not the case.

[jepler]

thank you