PYTHONPATH affects isolated builds

[sanderr]

Introduction

I've found that setting the PYTHONPATH environment variable affects isolated builds, to the point of breaking functionality in some cases. I'll describe steps to reproduce, and propose a fix, though I should disclaim that my experience with both this package and the PYTHONPATH environment variable are limited, so I might be missing something. This ticket follows from the assumption that setting the PYTHONPATH environment variable should not affect build behavior (as seems to be confirmed by https://github.com/pypa/build/issues/373#issuecomment-939545616). If this assumption is wrong, the ticket is invalid, though I'd be interested to know why this is the case.

Environment

Python 3.6.15. The same steps produce an error for 3.9 as well, but it is a different one (similar to the error in #266) so I'll create a separate ticket for that if it is agreed upon that this is a bug.

Steps to reproduce

# clone the sample project
git clone https://github.com/pypa/sampleproject.git
cd sampleproject
# create a new virtual environment
python3.6 -m venv .env
.env/bin/pip install -U pip build
# build the sample project with the env's libs dir in PYTHONPATH
PYTHONPATH=.env/lib/python3.6/site-packages/ .env/bin/python -m build --sdist

Expected behavior

An sdist is built

Observed behavior

* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools>=40.8.0, wheel)

Usage:   
  /tmp/build-env-a0xlzsq3/bin/python -m pip install [options] <requirement specifier> [package-index-options] ...
  /tmp/build-env-a0xlzsq3/bin/python -m pip install [options] -r <requirements file> [package-index-options] ...
  /tmp/build-env-a0xlzsq3/bin/python -m pip install [options] [-e] <vcs project url> ...
  /tmp/build-env-a0xlzsq3/bin/python -m pip install [options] [-e] <local project path> ...
  /tmp/build-env-a0xlzsq3/bin/python -m pip install [options] <archive url/path> ...

no such option: --use-pep517

Traceback (most recent call last):
  File "/home/sander/documents/projects/sampleproject/.env/lib/python3.6/site-packages/build/__main__.py", line 373, in main
    args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependency_check
  File "/home/sander/documents/projects/sampleproject/.env/lib/python3.6/site-packages/build/__main__.py", line 202, in build_package
    out = _build(isolation, builder, outdir, distribution, config_settings, skip_dependency_check)
  File "/home/sander/documents/projects/sampleproject/.env/lib/python3.6/site-packages/build/__main__.py", line 140, in _build
    return _build_in_isolated_env(builder, outdir, distribution, config_settings)
  File "/home/sander/documents/projects/sampleproject/.env/lib/python3.6/site-packages/build/__main__.py", line 108, in _build_in_isolated_env
    env.install(builder.build_system_requires)
  File "/home/sander/documents/projects/sampleproject/.env/lib/python3.6/site-packages/build/env.py", line 211, in install
    _subprocess(cmd)
  File "/home/sander/documents/projects/sampleproject/.env/lib/python3.6/site-packages/build/env.py", line 81, in _subprocess
    raise e
  File "/home/sander/documents/projects/sampleproject/.env/lib/python3.6/site-packages/build/env.py", line 78, in _subprocess
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/tmp/build-env-a0xlzsq3/bin/python', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', '-r', '/tmp/build-reqs-ont40sbg.txt']' returned non-zero exit status 2.

ERROR Command '['/tmp/build-env-a0xlzsq3/bin/python', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', '-r', '/tmp/build-reqs-ont40sbg.txt']' returned non-zero exit status 2.

Likely cause

IsolatedEnvBuilder sets up an isolated environment to perform the build using create_isolated_env_venv (virtualenv is not installed for this scencario). Package installation within this environment is then handled by _IsolatedEnvVenvPip.install, which passes Python's -I flag to ensure isolation: https://github.com/pypa/build/blob/90dbd8bd71e510c18b0a472e72f9e3503ee5a90f/src/build/env.py#L200-L209 However, this same flag is absent during setup: create_isolated_env_venv performs a pip install pip (simplified) to ensure a sufficiently recent pip is installed: https://github.com/pypa/build/blob/90dbd8bd71e510c18b0a472e72f9e3503ee5a90f/src/build/env.py#L275 As a result, the pip install pip during the set up of the environment still takes the PYTHONPATH environment variable into account, deciding that a matching pip is already installed. As a result the outdated pip in the isolated environment is not updated, which causes the failure seen above.

Proposed fix

Add the -I flag to both pip install commands in create_isolated_env_venv. I've confirmed that this resolves this issue.

[layday]

Thanks for the detailed report! This is (sort of) tracked in #373 and will be fixed by #361.

[sanderr]

Thanks for your response. I can confirm that the steps outlined above result in the desired behavior on that branch for Python 3.6. It still fails for 3.9 though. Should I create a separate ticket for that or will it be taken into account in the pull request?

[layday]

Are you seeing the same error or a different error?

[sanderr]

A different one

sander@arthur:~/documents/projects/sampleproject$ PYTHONPATH=.env/lib/python3.9/site-packages/ .env/bin/python -m build
 --sdist
* Creating isolated environment (venv)...

Traceback (most recent call last):
  File "/home/sander/documents/projects/build/src/build/__main__.py", line 149, in _handle_build_error
    yield
  File "/home/sander/documents/projects/build/src/build/__main__.py", line 380, in main
    built = build_call(
  File "/home/sander/documents/projects/build/src/build/__main__.py", line 206, in build_package
    out = _build(isolation, srcdir, outdir, distribution, config_settings, skip_dependency_check)
  File "/home/sander/documents/projects/build/src/build/__main__.py", line 141, in _build
    return _build_in_isolated_env(srcdir, outdir, distribution, config_settings)
  File "/home/sander/documents/projects/build/src/build/__main__.py", line 105, in _build_in_isolated_env
    with IsolatedEnvManager(_IsolatedEnv()) as env:
  File "/home/sander/documents/projects/build/src/build/env.py", line 290, in __enter__
    isolated_env.create(self._path)
  File "/home/sander/documents/projects/build/src/build/env.py", line 180, in create
    self._patch_up_venv(paths['purelib'])
  File "/home/sander/documents/projects/build/src/build/env.py", line 192, in _patch_up_venv
    cur_pip_version = next(
StopIteration

ERROR 

[sanderr]

I think venv.EnvBuilder(with_pip=True).create() needs to be run in isolation from the env vars as well, but I haven't dug deeper than that.

[layday]

Split the env var patch out of #361, see #406.

[sanderr]

I've run the above command with build at layday/override-leaky-env-vars (169b154) if that's what you mean. I get the same result.

[layday]

Yeah, I didn't change anything yet, I've just split it out so I can work on it separately.

I think venv.EnvBuilder(with_pip=True).create() needs to be run in isolation from the env vars as well, but I haven't dug deeper than that.

venv invokes ensurepip with the -I flag. This might seem like the right thing to do but ensurepip, in turn, invokes pip in a subprocess without -I. To work around this we'd either have to invoke venv in a(nother) subprocess or mutate the environment.

[sanderr]

Ah, I'd misunderstood your earlier comment. The venv invocation is indeed a bit trickier than I made it out to be. Do you consider this in scope for #406? I need to shift my attention elsewhere for now but let me know if I can help out somewhere still, I'll see if I can make the time.

[layday]

Fixing it is fairly trivial (and I've done so in #406) but all of these bodges are making my stomach turn.