sys._base_executable may not exist on posix venvs made with `--copies`

[vfazio]

Bug report

Python venv on POSIX environments creates a bin directory with:

• python
• python{VERSION_MAJOR}
• python(VERSION_MAJOR}.{VERSION_MINOR}

When executing python out of a venv made with the --copies argument specified (that is, it doesn't use symlinks), the calculation of base_executable will blindly append the basename of the executable to the value from the home key from pyvenv.cfg, meaning base_executable = $home_dir/basename(executable)

With recent changes made in 3.11, on Linux, the home key is determined by the base_executable of the initial python that created the venv. Even if the venv is "nested", home will still point to the topmost non-venv python bin path.

However, this can be problematic because source installations via make install only provide:

• python{VERSION_MAJOR}
• python(VERSION_MAJOR}.{VERSION_MINOR}

This means if "python" is invoked from a venv, the resultant base_executable may be an invalid file that applications may be relying upon via sys._base_executable (both venv and virtualenv use this value)

Most distributions do not distribute a version-less "python" as part of the divorce of python2 and python3 via guidance from PEP 394 (though some offer a convenience package to symlink it)

A suggested fix in previous discussions was to include a field in pyvenv.cfg specifying the base executable, but that may require updates to PEP 405 since there's only two documented keys for this file and other implementations such as virtualenv would want to conform to new specifications (virtualenv already stores a base-executable key so there could be a conflict)

In lieu of that, a simple stat to check for alternatives seems reasonable without impacting startup performance overly much.

Since this affects 3.11+, it'd be nice to see a fix backported to that branch.

Example:

vfazio@Zephyrus:~/development/cpython$ ./configure
vfazio@Zephyrus:~/development/cpython$ make -j12
vfazio@Zephyrus:~/development/cpython$ DESTDIR=/tmp/tmp.w9Bqq4Aibx/ make install
vfazio@Zephyrus:~/development/cpython$ cd /tmp/tmp.w9Bqq4Aibx/
vfazio@Zephyrus:~/development/cpython$ /tmp/tmp.w9Bqq4Aibx/usr/local/bin/python3 -V
Python 3.12.0a1+
vfazio@Zephyrus:~/development/cpython$ /tmp/tmp.w9Bqq4Aibx/usr/local/bin/python3 -m venv --clear --copies /tmp/tmp.307I606AVh/
vfazio@Zephyrus:~/development/cpython$ /tmp/tmp.307I606AVh/bin/python -c "import sys; print((sys._base_executable, sys.version))"
('/tmp/tmp.w9Bqq4Aibx/usr/local/bin/python', '3.12.0a1+ (heads/vfazio-base-exec-fallback:267e1040a9, Nov  6 2022, 15:16:06) [GCC 9.4.0]')
vfazio@Zephyrus:~/development/cpython$ test -e /tmp/tmp.w9Bqq4Aibx/usr/local/bin/python; echo $?
1
vfazio@Zephyrus:~/development/cpython$


root@fe1d04cd5d49:/tmp/tmp.SfKfnBG1SH# usr/local/bin/python3 -V
Python 3.11.0
root@fe1d04cd5d49:/tmp/tmp.SfKfnBG1SH# mktemp -d
/tmp/tmp.DDS5n25q4C
root@fe1d04cd5d49:/tmp/tmp.SfKfnBG1SH# usr/local/bin/python3 -m venv --copies --clear /tmp/tmp.DDS5n25q4C/
root@fe1d04cd5d49:/tmp/tmp.SfKfnBG1SH# /tmp/tmp.DDS5n25q4C/bin/python -c "import sys; print((sys._base_executable, sys.version))"
('/tmp/tmp.SfKfnBG1SH/usr/local/bin/python', '3.11.0 (main, Nov  5 2022, 11:43:28) [GCC 10.2.1 20210110]')
root@fe1d04cd5d49:/tmp/tmp.SfKfnBG1SH# test -e /tmp/tmp.SfKfnBG1SH/usr/local/bin/python; echo $?
1

Your environment

• CPython versions tested on: 3.11, main (3.12.0a1+)
• Operating system and architecture: Linux x86-64

related discussion and MRs: https://bugs.python.org/issue46028 https://github.com/python/cpython/pull/29041 https://github.com/python/cpython/pull/30144

related issues: https://github.com/pypa/virtualenv/issues/2440 https://github.com/python-poetry/poetry/issues/6940

• PR: gh-99206