Symlinks point to plain python3 executable instead of python3.x

[ericriff]

Issue

If you create an environment and you either don't specify a python version for it or specify the one which is set as default on the system, the symlinks point to python3 instead of python3.x. This leads to nasty bugs if you change your default python3 with, lets say, update-alternatives.

Case 1: Default python version is 3.6 and we create an env with 3.6. The symlink point to python3, therefore if you change the default version of python with update-alternatives, this environment gets broken.

$ python3 --version          
Python 3.6.12
$  python3 -m virtualenv .venv -p 3.6
created virtual environment CPython3.6.12.final.0-64 in 110ms
  creator CPython3Posix(dest=/home/eric/venvs-test/.venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/eric/.local/share/virtualenv)
    added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

$ tree .venv/bin
.venv/bin
├── activate
├── activate.csh
├── activate.fish
├── activate.ps1
├── activate_this.py
├── activate.xsh
├── easy_install
├── easy_install3
├── easy_install-3.6
├── easy_install3.6
├── pip
├── pip3
├── pip-3.6
├── pip3.6
├── python -> /usr/bin/python3
├── python3 -> python
├── python3.6 -> python
├── wheel
├── wheel3
├── wheel-3.6
└── wheel3.6

Case 2: Default python version is 3.6 and we create an env with 3.7. The symlink point to python3.7, therefor if you change the default version of python with update-alternatives, this environment will be fine. This should be the expected behavior.

$ python3 --version          
Python 3.6.12
$ python3 -m virtualenv .venv -p 3.7
created virtual environment CPython3.7.9.final.0-64 in 170ms
  creator CPython3Posix(dest=/home/eric/venvs-test/.venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/eric/.local/share/virtualenv)
    added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

$ tree .venv/bin
.venv/bin
├── activate
├── activate.csh
├── activate.fish
├── activate.ps1
├── activate_this.py
├── activate.xsh
├── easy_install
├── easy_install3
├── easy_install-3.7
├── easy_install3.7
├── pip
├── pip3
├── pip-3.7
├── pip3.7
├── python -> /usr/bin/python3.7
├── python3 -> python
├── python3.7 -> python
├── wheel
├── wheel3
├── wheel-3.7
└── wheel3.7

Case 3: Default python version is 3.7 and we create an env with 3.6. The symlink point to python3.6, which is what I expected on the case 1. This again produces an environment immune to the changes in the base system:

$ python3 --version                 
Python 3.7.9
$ python3 -m virtualenv .venv -p 3.6         
created virtual environment CPython3.6.12.final.0-64 in 90ms
  creator CPython3Posix(dest=/home/eric/venvs-test/.venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/eric/.local/share/virtualenv)
    added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
$ tree .venv/bin
.venv/bin
├── activate
├── activate.csh
├── activate.fish
├── activate.ps1
├── activate_this.py
├── activate.xsh
├── easy_install
├── easy_install3
├── easy_install-3.6
├── easy_install3.6
├── pip
├── pip3
├── pip-3.6
├── pip3.6
├── python -> /usr/bin/python3.6
├── python3 -> python
├── python3.6 -> python
├── wheel
├── wheel3
├── wheel-3.6
└── wheel3.6

Environment

Provide at least:

• OS: Ubuntu 18
• virtualenv: 20.0.33

I hope this is detailed enough and that it is not a duplicated issue. I couldn't find another one like this.

Thanks!

[gaborbernat]

I'd guess this is one of thos works as expected and as designed. virtualenvs do not survive replacing your system python with another path.

Not sure how we can do better per se here, without in general degrading performance. Instead of accepting the first found python interpreter executable we'd need to discover all, and then decide in between them. But now interpreter discovery always takes much longer 🤔 which I'd like to avoid.

Can you provide a Docker image reproducible?

[ericriff]

Trying to resolve where python3 shouldn't be enough? So if python3 matches de criteria, then try to resolve the symlink and skip the middleman. The performance hit would be next to nothing

[gaborbernat]

We already should resolve the middleman, so here must be something else. The Python interpreter must be reporting the symlink is the host python (resolved). Please provide a reproducible for docker so we can better understand the issue.

[ericriff]

I have no experience with docker. I just have multiple python versions installed with apt and then switch between them with update-alternatives. I'll see if I can came up with something.

[ericriff]

You can replicate it with this dockerfile:

FROM ubuntu:18.04

RUN apt-get update && apt-get upgrade -y && apt-get clean

# Install python 3.6 (Ubuntu 18 default) and 3.7 
RUN apt-get install -y python3.7 python3.7-dev python3.7-distutils python3.6 python3.6-dev python3.6-distutils

RUN apt-get install -y curl tree

# Install alternatives
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 2

# Set python 3.6 as the default python
RUN update-alternatives --set python3 /usr/bin/python3.6

# Install pip
RUN curl -s https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python3 get-pip.py --force-reinstall && \
    rm get-pip.py

# Install virtualenv
RUN python3 -m pip install virtualenv

Just do:

$ docker build -t virtualenvbug .
$ docker run -it --rm virtualenvbug

$ virtualenv .venv36 -p 3.6
$ virtualenv .venv37 -p 3.7

$ tree .venv36/bin
$ tree .venv37/bin

And you will see that the virtualenvironmet using the default version of python, that is, python3.6, will have a symlink to python3 and not python3.6. The other one works just fine. python -> /usr/bin/python3 python -> /usr/bin/python3.7

[gaborbernat]

Thanks can take that to investigate and use to validate the fix too. Feel free to start work on that if you have time, otherwise it hits the backlog, and hopefully someone picks it up eventually 👍

[ericriff]

I don't think I can fix this (at least in a non-hacky way) since I just started using virtualenv, but I'll take a look. I though this was an issue with Poetry tbh, but after some digging I pinpointed it here. They are moving to avoiding synlinks to get around this. https://github.com/python-poetry/poetry/pull/3157

[ericriff]

As a quick test, on discovery/py_info.py, on _resolve_to_system() I forced:

target.executable = os.path.realpath(start_executable)
target.system_executable = os.path.realpath(target.system_executable)

Being the second line the one that does the trick for me.

Going a little bit further down the rabbit hole, I came across _fast_get_system_executable() which has a comment made by you @gaborbernat which says:

        # if we're not in a virtual environment, this is already a system python, so return the original executable
        # note we must choose the original and not the pure executable as shim scripts might throw us off

When you say pure executable are you talking about the actual executable and not the symlink?

[gaborbernat]

Oh, pure executable means the actual python executable (what we get from sys.executable/sys.base_executable), and not potentially any random shim scripts, e.g. for when people do something like:

cat /usr/bin/python3
!#/bin/bash
export MAGICAL_SETUP="ok"
/magic/python3 "$@"

[gaborbernat]

As this is 3 years old without updates closing it.