venv initialisation fails when build is installed in user library

[deprekate]

Thank you for providing feedback on Python packaging!

To help us help you, please fill out as much of the following as you can. If a question is not relevant, feel free to skip it.

1. What is your operating system and version? This issue is for trying to get it run on OSX Big Sur (I've commented on the other [closed] issues with the errors I get when trying on various linux machines)

2. What is your Python version? Python 3.8.2

3. What version of pip do you have? pip 21.1.2 from /Users/katelyn/Library/Python/3.8/lib/python/site-packages/pip (python 3.8)

4. If following an online tutorial or guide, please provide a link to the page or section giving you trouble:

https://packaging.python.org/tutorials/packaging-projects/

5. Could you describe your issue in as much detail as possible?

$ python3 -m build
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 236, in <module>
    main(sys.argv[1:], 'python -m build')
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 228, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependency_check)
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 105, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 63, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/env.py", line 92, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/env.py", line 218, in _create_isolated_env_venv
    executable, script_dir, purelib = _find_executable_and_scripts(path)
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/env.py", line 257, in _find_executable_and_scripts
    raise RuntimeError('Virtual environment creation failed, executable {} missing'.format(executable))
RuntimeError: Virtual environment creation failed, executable /usr/local/bin/python missing

I guess pypa uses hardcoded paths? So I tried to symlink the correct structure:

$ sudo ln -s /usr/local/bin/python /usr/local/bin/python3

Just more errors down the rabbit hole:

$ python3 -m build
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 236, in <module>
    main(sys.argv[1:], 'python -m build')
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 228, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependency_check)
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 105, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 63, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/env.py", line 92, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/Users/katelyn/Library/Python/3.8/lib/python/site-packages/build/env.py", line 221, in _create_isolated_env_venv
    pip_distribution = next(iter(metadata.distributions(name='pip', path=[purelib])))
StopIteration

[layday]

Thank you for the report. I believe the issue here is that because you installed build in the user library, the user scheme ends up being used when retrieving the venv paths. We should be overriding the scheme on https://github.com/pypa/build/blob/781c0f1894463c587a0c9f9738c7b8b2f71bb4bc/src/build/env.py#L249. In the meantime, you can install build in a virtual environment and invoke it from there.

[layday]

Actually, it's the osx_framework_library scheme that's being used but the net effect is the same. I gather that even with framework builds of Python on macOS, venvs are created according to posix_prefix. Therefore, the scheme should always be posix_prefix on macOS and Linux and nt on Windows.

[uranusjr]

Technically a venv can be created with osx_framework_library, but existing implementations all use posix_prefix exclusively. The most bullet-proof way would be to use sysconfig.get_default_scheme() on Python 3.10 and sysconfig._get_default_scheme() for previous versions.

https://docs.python.org/3.10/library/sysconfig.html#sysconfig.get_default_scheme

[layday]

Apple patch _get_default_scheme to return osx_framework_library; Apple's Python will have to be special-cased.

[uranusjr]

Urgh, in that case I guess it's best to hard-code posix_prefix then, since that's what all virtual environment implementations do. We can worry about other schemes if someone really implements PEP 405 with another install scheme.

[gaborbernat]

Or you know, just use virtualenv that has this patch-ed and handled 😎

@deprekate do you still get the issue when you install build[virtualenv]?

[layday]

Unfortunately, virtualenv has also fallen victim to Apple's machinations: https://github.com/pypa/build/issues/294.

[uranusjr]

I think this issue should be split into two parts:

1. Should this be considered a downstream bug (Apple in this case) that sysconfig._get_default_scheme() does not accurately reflect the scheme in sys.prefix? (We should make Apple fix this if that’s the case.)
2. If not, should there be a mechanism that can be used for tools introspect what scheme a prefix is using? What should that mechanism be, a function, or a config value (maybe in `pyvenv.cfg)?

This probably needs to be escalated to the broader packaging community and potentially CPython core devs, depending on how people think we solution should be.

[layday]

Apple's sysonfig._get_default_scheme does accurately reflect sys.prefix. The issue here is that the sys.prefix of the running interpreter (i.e. the platform scheme) is different from the virtual environment's. Apple falls foul of PEP 405, which says:

In order to allow [snip] Python package managers to install packages into the virtual environment the same way they would install into a normal Python installation, and avoid special-casing virtual environments in sysconfig beyond using sys.base_prefix in place of sys.prefix where appropriate, the internal virtual environment layout mimics the layout of the Python installation itself on each platform [...] This approach explicitly chooses not to introduce a new sysconfig install scheme for venvs. Rather, by modifying sys.prefix we ensure that existing install schemes which base locations on sys.prefix will simply work in a venv. Installation to other install schemes (for instance, the user-site schemes) whose paths are not relative to sys.prefix, will not be affected by a venv at all.

Apple breaks the core assumption that the platform scheme and virtual environment scheme are the same. In sysconfig, it detects whether Python is running inside a virtual environment; if it does, it returns posix_prefix, otherwise it returns osx_framework_library, a custom Apple scheme not found in a standard Python installation.

---

Seeing as not everybody has access to macOS, this is what Apple's _get_default_scheme looks like:

_INSTALL_SCHEMES = {
    ...,
    'osx_framework_library': {
        'stdlib': '{installed_base}/lib/python{py_version_short}',
        'platstdlib': '{platbase}/lib/python{py_version_short}',
        'purelib': '/Library/Python/{py_version_short}/site-packages',
        'platlib': '/Library/Python/{py_version_short}/site-packages',
        'include': '/Library/Python/{py_version_short}{abiflags}/include',
        'platinclude': '/Library/Python/{py_version_short}{abiflags}/include',
        'scripts': '/usr/local/bin',
        'data': '/Library/Python/{py_version_short}',
        },
}

def _use_darwin_global_library():
    if sys.platform == 'darwin' and sys._framework:
        prefix = os.path.dirname(sys.prefix)
        framework_versions = sys._framework + '.framework/Versions'
        if prefix.endswith(framework_versions):
            frameworks = os.path.dirname(os.path.dirname(prefix))
            if (frameworks in
                    ['/System/Library/Frameworks',
                     '/AppleInternal/Library/Frameworks',
                     '/Library/Frameworks',
                     '/Library/Developer/CommandLineTools/Library/Frameworks']):
                return True
            if frameworks.endswith('.app/Contents/Developer/Library/Frameworks'):
                return True
    return False

def _get_default_scheme():
    if _use_darwin_global_library():
        return 'osx_framework_library'
    if os.name == 'posix':
        # the default scheme for posix is posix_prefix
        return 'posix_prefix'
    return os.name

[uranusjr]

Apple's sysonfig._get_default_scheme does accurately reflect sys.prefix. The issue here is that the sys.prefix of the running interpreter (i.e. the platform scheme) is different from the virtual environment's.

You probably meant sys.base_prefix; because sys.prefix points to the virtual environment’s root when the interpreter is inside one. I think we actually agree here; Apple is doing things wrong because sysconfig._get_preferred_scheme() does not reflect the scheme of sys.prefix when the interpreter is running in a virtual environment.

Apple falls foul of PEP 405, which says:

The PEP wording is a bit ambiguous to me, to be honest. It is kind of speaking under the assumption that each platform has one (default) installation layout, which is not true for downstream distributed Pythons. As far as I am aware, almost all Linux distribution-distributed Python installations would unfortunately fail under your interpretation, including Debian, Red Hat, etc., because they all use a vendor-defined default scheme, but uses posix_prefix in virtual environments. What they do right is patching sysconfig (and distutils) correctly so the interpreter reports its default scheme according to the its runtime environment, including the surrounding virtual environment.

[layday]

You probably meant sys.base_prefix; because sys.prefix points to the virtual environment’s root when the interpreter is inside one.

sysconfig._get_default_scheme returns the correct scheme provided that it is called from inside the venv - the return value of sysconfig._get_default_scheme is always appropriate for the sys.prefix of the running interpreter. This is why we're only hearing of this issue now, because when build is installed in a venv, sysconfig._get_default_scheme will return posix_prefix and that is appropriate for the venv that build itself creates for build isolation. Of course, that's not what sysconfig.(_)get_default_scheme should do as it is now publicly documented.

It is kind of speaking under the assumption that each platform has one (default) installation layout, which is not true for downstream distributed Pythons.

AFAIK Linux distros do not accomplish this by varying the default scheme although it is conceivable that they might do that in Python 3.10. I think it'd be informative to know how they approach this in some more detail - it'll help with what we decide to do here.

[uranusjr]

sysconfig._get_default_scheme returns the correct scheme provided that it is called from inside the venv - the return value of sysconfig._get_default_scheme is always appropriate for the sys.prefix of the running interpreter.

Oh! So I misunderstood the issue entirely. Thanks for the correction. So to make sure I’m on the correct page, Apple’s system Python returns osx_framework_library when called in the default prefix, and posix_prefix when inside a virtual environment. Assuming that’s correct, I don’t think it’s doing anything wrong here—at least not in the practical sense. It may be argued it is not following PEP 405, but I think the community would probably opt to amend PEP 405 to follow that behaviour. So pipx might not have much choice but to fix the logic to fetch the information from the virtual environment instead.

I proposed a while ago that a virtual environment tool should record what scheme it is creating the virtual environment as in pyvenv.cfg, but that got shot down with roughly “why not just do a subprocess call to get that information”. So I guess that should be the solution here.

[layday]

Assuming that’s correct

That is correct.

So pipx might not have much choice but to fix the logic to fetch the information from the virtual environment instead.

I would instead propose to copy the default _get_default_scheme and pass its return value to get_paths(scheme=...). We know that the venv module creates a venv with the nt scheme on Windows and posix_prefix everywhere else whatever sysconfig might say.

[layday]

Right, PyPy3 uses a scheme for venvs which diverges from posix_prefix, so we must necessarily rely on sysconfig. I find starting up a subprocess just to find what scheme's being used very unsatisfactory.

[uranusjr]

It definitely is unsatisfactory. Maybe it’s time to bring up the topic putting scheme in pyvenv.cfg again. Would you want to post this to python-dev? I feel you’re in a better position than myself to describe the issue and make a better case to the proposal.

[gaborbernat]

I proposed a while ago that a virtual environment tool should record what scheme it is creating the virtual environment as in pyvenv.cfg, but that got shot down with roughly “why not just do a subprocess call to get that information”. So I guess that should be the solution here.

As a virtualenv maintainer, I'd accept such PR, though I have no powers over venv...