virtualenv with --system-site-packages breaks pip's build isolation

[pganssle]

It seems that if you create a virtualenv with --system-site-packages, the system packages (but not the user packages) will be on the PYTHONPATH in the PEP 517 isolated build environment that pip creates (it does not effect python -m pep517.build), and it seems they will be on there with higher precedence than the requirements installed in the build environment. It only affects pip >= 19.0.0.

The most common way I've seen this cause problems is with the new setuptools.build_meta:__legacy__ backend. Since the most recent version of pip requires a recent version of setuptools, if you have an older version of setuptools installed on the system, pip install will fail due to the missing setuptools.build_meta:__legacy__ backend. It is possible to reproduce this by crafting a deliberately bad package and installing it, then creating a wheel for a newer version of the package (which allowed me to test that this failure was actually introduced with pip==19.0.0), but for the MWE we can stick with setuptools.

To reproduce, create a package like this:

cd /tmp
mkdir demo
cd demo
mkdir badpkg

touch badpkg/pyproject.toml
echo 'from setuptools import setup; setup(name="foo", version="0.1.0")' \
    > badpkg/setup.py

Then install an older version of setuptools on your system (can't be --user or in the virtualenv), pip install 'setuptools < 40.8.0' (I did this in a pyenv environment). If your system already has an older version of setuptools on it, you're already good.

Next create a virtualenv with --system-site-packages and activate it:

virtualenv venv --system-site-packages
source venv/bin/activate

Finally try to install badpkg (pip wheel or pip install -t tmp also works to demonstrate the problem):

pip install ./badpkg

You should get a traceback the ends like this:

AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

At first I thought this was because the affected packages had too loose bounds on build-system.requires (e.g. requires=["setuptools"]) and that pip was failing to install a more recent version in the isolated environment, but this bug still occurs even if you specify "setuptools>=40.8.0", so I believe it's not a problem at install-time, it's a problem with the path resolution at build time.

At the moment it's unclear if this is a problem with pip or virtualenv, but since it does not affect python -m pep517.build, I'm reporting it here. It could be a combination of both.

CC: @gaborbernat

[pganssle]

Per my Bloomberg colleague @lkollar, it seems that the problem is that pip creates a custom sitecustomize.py file in order to inject itself into the build environment.

Looks to me like it's just blacklisting the system packages with distutils.get_python_lib, which doesn't include the system path added by virtualenv. Seems like pep517.envbuild.BuildEnvironment doesn't have this problem - maybe switch over to using that? Or was there a reason to do it this way?

[boegel]

I'm also seeing this problem, when installing cryptography 2.5 using Python 3.7.2, even though I'm using the latest setuptools (40.8.0).

What I don't understand is why I'm seeing this even though I'm using pip install --no-build-isolation .... How is that possible if the problem is related to pip's build isolation feature?

[boegel]

I should also mention that I'm not using virtualenv at all (although maybe somehow the cryptography installation procedure does that somehow?).

[pganssle]

@boegel Can you create a minimal-working example that reproduces this, and include your pip versions? I've never known anyone to be able to trigger this without virtualenv --system-site-packages.

[boegel]

@pganssle Does this suffice?

Note that this is a self-built Python 3.7.2 from source, in a custom location, not sure if that's relevant to the problem...

$ python -V
Python 3.7.2

$ pip -V
pip 19.0.3 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip (python 3.7)

$ easy_install --version
setuptools 40.8.0 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.8.0-py3.7.egg (Python 3.7)

$ mkdir -p /tmp/$USER/pip_debug
$ cd /tmp/$USER/pip_debug

$ curl -OL https://files.pythonhosted.org/packages/69/ed/5e97b7f54237a9e4e6291b6e52173372b7fa45ca730d36ea90b790c0059a/cryptography-2.5.tar.gz

$ tar xfz cryptography-2.5.tar.gz

$ cd cryptography-2.5

$ pip install --prefix=$PWD --no-deps  --ignore-installed  --no-build-isolation  .
Processing /tmp/user/pip_debug/cryptography-2.5
    Preparing wheel metadata ... error
    Complete output from command /prefix/Python/3.7.2-GCCcore-8.2.0/bin/python /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpkwc5sc4e:
    Traceback (most recent call last):
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 62, in prepare_metadata_for_build_wheel
        backend = _build_backend()
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
        obj = getattr(obj, path_part)
    AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

    ----------------------------------------
Command "/prefix/Python/3.7.2-GCCcore-8.2.0/bin/python /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpkwc5sc4e" failed with error code 1 in /tmp/pip-req-build-q9knj58j

This is on CentOS 7.6.1810, with only python 2.x installed on the system. We do have an old setuptools in the OS as well, but I think that one is irrelevant since that's only for Python 2.

$ rpm -qa | grep ^python[0-9]*-[0-9]
python-2.7.5-76.el7.x86_64
$ rpm -qa | grep setuptools
python-setuptools-0.9.8-7.el7.noarch

[boegel]

I should also mention that --no-use-pep517 fixes the problem for me.

[pfmoore]

pip 19.0.3 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip (python 3.7)

Looks like your pip (and your setuptools) have been installed as eggs, presumably using easy_install. This is obsolete, can you try with a copy of pip installed using pip?

[boegel]

@pfmoore Same problem when pip and setuptools were installed with pip, when using latest cryptography from PyPI. Note that there's no other Python 3 installation on the system (and hence no other setuptools either).

$ which python
/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python
$ python -V
Python 3.7.2

$ pip -V
pip 19.0.3 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip (python 3.7)

$ easy_install --version
setuptools 40.8.0 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages (Python 3.7)

$ mkdir -p /tmp/$USER/pip_debug
$ cd /tmp/$USER/pip_debug
$ curl -OL https://files.pythonhosted.org/packages/07/ca/bc827c5e55918ad223d59d299fff92f3563476c3b00d0a9157d9c0217449/cryptography-2.6.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  480k  100  480k    0     0   755k      0 --:--:-- --:--:-- --:--:--  754k

$ tar xfz cryptography-2.6.1.tar.gz
$ cd cryptography-2.6.1

$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  .
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
    Preparing wheel metadata ... error
    Complete output from command /prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_:
    Traceback (most recent call last):
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 62, in prepare_metadata_for_build_wheel
        backend = _build_backend()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
        obj = getattr(obj, path_part)
    AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

    ----------------------------------------
Command "/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_" failed with error code 1 in /tmp/pip-req-build-s_w63g1g

Works fine when using --no-use-pep517:

$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  . --no-use-pep517
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
Building wheels for collected packages: cryptography
  Building wheel for cryptography (setup.py) ...
...

[pfmoore]

module 'setuptools.build_meta' has no attribute 'legacy'

That indicates you're not picking up setuptools 40.8.0 - see here

Try running

import setuptools.build_meta
print(setuptools.build_meta.__legacy__)

You should get something like <setuptools.build_meta._BuildMetaLegacyBackend object at 0x000000000401B2E8>. If you're not, then there's something wrong with your system setuptools. If you are, your build environment is not picking up the same packages as your system Python - in spite of your assertion that there's only the one setuptools installed on your system :-(

Also, the fact that you're using --prefix may be relevant here - I don't know how that interacts (if at all) with build environments. Can you reproduce this issue without --prefix?

[gaborbernat]

@pfmoore if virtualenv inherits the system site package (and that has a setuptoools before 40.8.0) the pip install of the setuptools build dependency is ignored (as pip does not check the version of what is installed unless -U is passed, and says any setuptools satisfies the install requirement); therefore once pip tries to get the build backend the above failure is thrown. I don't think there needs to be anything wrong with system python to fall into this issue. pip should make sure to never-ever create an isolated build environment that inherits from the system site package, which is not the case at the moment.

[pfmoore]

if virtualenv inherits the system site package

Presumably when --no-build-isolation is specified? An isolated build ignores the existing installation.

(and that has a setuptoools before 40.8.0)

Well, the point of --no-build-isolation is that the user is agreeing to set up the correct build environment, so that seems like it's a simple case of user error.

However, in this case @boegel is claiming that he has setuptools 40.8.0 installed, and no other setuptools is present. If that's the case, and his setuptools doesn't contain build_meta.__legacy__, then that setuptools installation is somehow broken, because 40.8.0 does contain that backend.

My suspicion is that @boegel is actually mistaken, and there's another copy of setuptools lingering around somewhere, which is getting picked up and is confusing things. The other possibility is that there's a bug somewhere - but if that is the case, then we're currently struggling to reduce the example to something that can easily reproduce/demonstrate that bug.

pip should make sure to never-ever create an isolated build environment that inherits from the system site package

Well, yes, that's sort of the definition of "isolated" :-) But I'm not sure how relevant that is in this case, where --no-build-isolation is set.

which is not the case at the moment

That's what @pganssle is saying in the original post in this thread, yes. But I'm responding to @boegel, who is claiming that he can reproduce the problem without a virtualenv, using --no-build-isolation.

If @boegel's problem turns out to be real, then I don't think it's related to this one (there's too many differences in the steps to reproduce it). So it should probably be a separate issue. But I'd be interested to understand why he thinks it is the same issue...

[boegel]

Dug a little further, it looks it's indeed a case of a broken setuptools installation, in some sense...

First, I should clarify my statement w.r.t. not having another setuptools installed. I should clarify that I mean that there's no other setuptools installed for Python 3.x. There is one for Python 2 (which is available in the OS), I'm assuming that one will never be picked up by Python 3.

That said, I'm pretty sure the right setuptools is being picked up:

$ python -c "import setuptools; print(setuptools.__file__); print(setuptools.version.__version__)"
/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools/__init__.py
40.8.0

Yet:

$ python -c "import setuptools.build_meta; print(setuptools.build_meta.__file__); print(setuptools.build_meta.__legacy__)"
/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools/build_meta.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

So I do have setuptools 40.8.0, but no __legacy__ in setuptools.build_meta?!

This sheds some light on it:

ls -ld /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools*
drwxr-xr-x 6 myuser myuser 4096 Mar  7 18:48 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:56 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.6.2.dist-info
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:57 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.8.0-py3.7.egg-info
drwxr-xr-x 3 myuser myuser 4096 Mar  7 11:58 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools_scm
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:58 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools_scm-3.2.0.dist-info

The setuptools-40.6.2.dist-info looks very suspicious, it seems like I somehow ended up with a frankenbuild of setuptools-40.6.2 (which doesn't have __legacy__ in setuptools.build_meta yet) and setuptools-40.8.0?! Ugh...

I got redirect to this issue via https://github.com/pypa/setuptools/issues/1694, which shows the exact same error I'm running into, so I naively assumed it's the same problem.

It seems in my case it's indeed because of a corrupted setuptools though, my apologies for the noise...

[pfmoore]

No problem, glad we got to the bottom of it!

[Lawouach]

Hey,

I'm running into this issue as well and I'm thoroughly confused. As a mere user, it looks totally broken with no clear path of understanding how I'm doing anything wrong.

[Lawouach]

Well, having said that. I ended up removing the system-wide setuptools package and now local pip doesn't complain.

[tgs]

I'm encountering the same issue, I think. My organization defaults virtualenvs to --system-site-packages, and the globally installed setuptools is 40.4.3. I cloned the pip git repo, and then:

In a system-site-packages virtualenv, pip install ./pip/ gives the AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'.

In a virtualenv with no-site-packages, pip install ./pip/ works correctly.

I think I agree with @gaborbernat that it would be better for "build isolation" to include ignoring --system-site-packages. Or maybe there's at least some way to give a better error message?

[webknjaz]

FTR I'm hitting this with tox + system-site-packages with setuptools>=40.8.0 in pyproject.toml under Python 2.7.15. I was actually hitting pypa/setuptools#1136 originally (which I knew was fixed a while back) and only after some time of playing around with removing pyproject.toml and then just playing with --no-build-isolation I've found this one...

Couldn't figure it out for a while because it was only reproducible in Travis CI which drops you into a virtualenv from the beginning automatically. Had to request debug access to their VMs to nail it down...

[webknjaz]

Okay, now I'm hitting this in https://github.com/rtfd/readthedocs.org/issues/5572 somehow.

[webknjaz]

They use explicit --no-site-packages: https://readthedocs.org/projects/octomachinery/builds/8869252/

[webknjaz]

@pganssle it looks like it's time to remove --system-site-packages from the title. I've already hit two instances of the bug with --no-site-packages.

[pganssle]

@webknjaz How does it manifest with --no-site-packages? That may even be a distinct bug.

[webknjaz]

@pganssle it looks the same to me: setuptools fails to use a newer feature because it's not the version of setuptools I specified in pyproject.toml

[webknjaz]

@pganssle one is https://readthedocs.org/projects/octomachinery/builds/8869252/ (pure virtualenv) another is https://travis-ci.com/sanitizers/octomachinery/jobs/190175604#L401-L407 (tox). Both don't have system site packages.

[webknjaz]

Yea, it's another missing feature hit but the root of the issue is still the same.

[pganssle]

I just meant we already know the cause of the problem with --system-site-packages , can you verify if it's the same mechanism here?

[webknjaz]

@pganssle can you point me to the comment with verification plan, please? I've scanned through and it's not obvious which one to follow...

[pganssle]

@webknjaz Just this: https://github.com/pypa/pip/issues/6264#issuecomment-463229977

TBH I remember verifying this by setting a breakpoint or printing something, but I guess at the time it seemed obvious what to do once you knew the cause 😅.

[webknjaz]

I'd need more precise instructions because debugging this with all subprocesses and stuff is PITA and unfortunately I don't have time for this right now... Maybe later.

[rkaplan]

Hitting this issue when installing scikit-learn inside a --system-site-packages virtualenv that uses Python 3.6.8 on Ubuntu 18.04. My setuptools version is 41.0.1.

[naisy]

I also hit this problem. In my case, downgrade pip. it works.

pip install --upgrade pip==19.0
pip install numpy