virtualenv --system-site-packages is ignored

Issue

Previously, the --system-site-packages flag (https://virtualenv.pypa.io/en/latest/cli_interface.html#section-creator) allowed a virtualenv to inherit Python packages from the parent Python installation.

Now, however, the virtualenv is trying to reinstall packages that should be available from the system. This can be observed, for example, by using a dependency that requires special handling to install, such as detectron2. (Of course this can also be observed by manually installing any non-default version of any package, such as from a branch or a local version installed with pip install --editable; virtualenv will ignore --system-site-packages and install the default version.)

A full example can be found at https://github.com/dHannasch/tox-sitepackages-example.

The peculiar thing is that everything worked perfectly as of July30 2:18pm; that is, --system-site-packages properly inherited the installed detectron2 from the environment. And yet, reverting to earlier versions of virtualenv (and even pip, since I noticed there was a new version of pip August 4) does not seem to help. (This is difficult to show, since that other library is complicated with a lot of other things going on, but it's striking that re-running the exact same CI job on the exact same git commit fails where it passed before. I presume the difference is some non-version-pinned library I haven't thought of, but it's apparently not virtualenv itself; some change in another library has broken --system-site-packages.)

Environment

This can be easily seen just running on Travis. https://github.com/dHannasch/tox-sitepackages-example/blob/master/.travis.yml shows everything that gets installed.

pip list of the host python where virtualenv is installed can also be seen in the Travis log.

Output of the virtual environment creation

Make sure to run the creation with -vvv --with-traceback:

$ python -m virtualenv -vvv --with-traceback --system-site-packages --no-download py38

72 setup logging to NOTSET [DEBUG report:43]

73 created app data folder /home/travis/.local/share/virtualenv [DEBUG __init__:30]

96 find interpreter for spec PythonSpec(path=/home/travis/virtualenv/python3.8.0/bin/python) [INFO builtin:44]

96 discover exe for PythonInfo(spec=CPython3.8.0.final.0-64, exe=/home/travis/virtualenv/python3.8.0/bin/python, platform=linux, version='3.8.0 (default, Oct 15 2019, 17:49:23) \n[GCC 5.4.0 20160609]', encoding_fs_io=utf-8-utf-8) in /opt/python/3.8.0 [DEBUG py_info:370]

96 filesystem is case-sensitive [DEBUG info:28]

98 Attempting to acquire lock 140630008408192 on /home/travis/.local/share/virtualenv/py_info/1/672c911cc59b7d5c5a3bbe4f1659fcb850a772217f5033cdad75c20df6b7549e.lock [DEBUG filelock:270]

98 Lock 140630008408192 acquired on /home/travis/.local/share/virtualenv/py_info/1/672c911cc59b7d5c5a3bbe4f1659fcb850a772217f5033cdad75c20df6b7549e.lock [INFO filelock:274]

98 get interpreter info via cmd: /opt/python/3.8.0/bin/python3.8 /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:87]

162 wrote python info of /opt/python/3.8.0/bin/python3.8 at /home/travis/.local/share/virtualenv/py_info/1/672c911cc59b7d5c5a3bbe4f1659fcb850a772217f5033cdad75c20df6b7549e.json [DEBUG via_disk_folder:159]

162 Attempting to release lock 140630008408192 on /home/travis/.local/share/virtualenv/py_info/1/672c911cc59b7d5c5a3bbe4f1659fcb850a772217f5033cdad75c20df6b7549e.lock [DEBUG filelock:315]

162 Lock 140630008408192 released on /home/travis/.local/share/virtualenv/py_info/1/672c911cc59b7d5c5a3bbe4f1659fcb850a772217f5033cdad75c20df6b7549e.lock [INFO filelock:318]

162 proposed PythonInfo(spec=CPython3.8.0.final.0-64, system=/opt/python/3.8.0/bin/python3.8, exe=/home/travis/virtualenv/python3.8.0/bin/python, platform=linux, version='3.8.0 (default, Oct 15 2019, 17:49:23) \n[GCC 5.4.0 20160609]', encoding_fs_io=utf-8-utf-8) [INFO builtin:50]

163 accepted PythonInfo(spec=CPython3.8.0.final.0-64, system=/opt/python/3.8.0/bin/python3.8, exe=/home/travis/virtualenv/python3.8.0/bin/python, platform=linux, version='3.8.0 (default, Oct 15 2019, 17:49:23) \n[GCC 5.4.0 20160609]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:52]

198 create virtual environment via CPython3Posix(dest=/home/travis/build/dHannasch/tox-sitepackages-example/py38, clear=False, global=True) [INFO session:52]

198 create folder /home/travis/build/dHannasch/tox-sitepackages-example/py38/bin [DEBUG _sync:25]

199 create folder /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages [DEBUG _sync:25]

199 write /home/travis/build/dHannasch/tox-sitepackages-example/py38/pyvenv.cfg [DEBUG pyenv_cfg:34]

199 	home = /opt/python/3.8.0 [DEBUG pyenv_cfg:38]

199 	implementation = CPython [DEBUG pyenv_cfg:38]

199 	version_info = 3.8.0.final.0 [DEBUG pyenv_cfg:38]

199 	virtualenv = 20.0.27 [DEBUG pyenv_cfg:38]

199 	include-system-site-packages = true [DEBUG pyenv_cfg:38]

199 	base-prefix = /opt/python/3.8.0 [DEBUG pyenv_cfg:38]

199 	base-exec-prefix = /opt/python/3.8.0 [DEBUG pyenv_cfg:38]

199 	base-executable = /opt/python/3.8.0/bin/python3.8 [DEBUG pyenv_cfg:38]

200 symlink /opt/python/3.8.0/bin/python3.8 to /home/travis/build/dHannasch/tox-sitepackages-example/py38/bin/python [DEBUG _sync:44]

200 create virtualenv import hook file /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/_virtualenv.pth [DEBUG api:94]

200 create /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/_virtualenv.py [DEBUG api:97]

201 ============================== target debug ============================== [DEBUG session:54]

202 debug via /home/travis/build/dHannasch/tox-sitepackages-example/py38/bin/python /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/virtualenv/create/debug.py [DEBUG creator:209]

201 {

  "sys": {

    "executable": "/home/travis/build/dHannasch/tox-sitepackages-example/py38/bin/python",

    "_base_executable": "/home/travis/build/dHannasch/tox-sitepackages-example/py38/bin/python",

    "prefix": "/home/travis/build/dHannasch/tox-sitepackages-example/py38",

    "base_prefix": "/opt/python/3.8.0",

    "real_prefix": null,

    "exec_prefix": "/home/travis/build/dHannasch/tox-sitepackages-example/py38",

    "base_exec_prefix": "/opt/python/3.8.0",

    "path": [

      "/opt/python/3.8.0/lib/python38.zip",

      "/opt/python/3.8.0/lib/python3.8",

      "/opt/python/3.8.0/lib/python3.8/lib-dynload",

      "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages",

      "/opt/python/3.8.0/lib/python3.8/site-packages"

    ],

    "meta_path": [

      "<class '_virtualenv._Finder'>",

      "<class '_frozen_importlib.BuiltinImporter'>",

      "<class '_frozen_importlib.FrozenImporter'>",

      "<class '_frozen_importlib_external.PathFinder'>"

    ],

    "fs_encoding": "utf-8",

    "io_encoding": "utf-8"

  },

  "version": "3.8.0 (default, Oct 15 2019, 17:49:23) \n[GCC 5.4.0 20160609]",

  "makefile_filename": "/opt/python/3.8.0/lib/python3.8/config-3.8-x86_64-linux-gnu/Makefile",

  "os": "<module 'os' from '/opt/python/3.8.0/lib/python3.8/os.py'>",

  "site": "<module 'site' from '/opt/python/3.8.0/lib/python3.8/site.py'>",

  "datetime": "<module 'datetime' from '/opt/python/3.8.0/lib/python3.8/datetime.py'>",

  "math": "<module 'math' from '/opt/python/3.8.0/lib/python3.8/lib-dynload/math.cpython-38-x86_64-linux-gnu.so'>",

  "json": "<module 'json' from '/opt/python/3.8.0/lib/python3.8/json/__init__.py'>"

} [DEBUG session:55]

235 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/travis/.local/share/virtualenv) [INFO session:59]

239 wrote embed update of distribution setuptools at /home/travis/.local/share/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:159]

240 wrote embed update of distribution pip at /home/travis/.local/share/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:159]

240 wrote embed update of distribution wheel at /home/travis/.local/share/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:159]

244 triggered periodic upgrade of setuptools==49.2.0 (for python 3.8) via background process having PID 3702 [INFO periodic_update:191]

248 got embed update of distribution setuptools from /home/travis/.local/share/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:133]

259 triggered periodic upgrade of wheel==0.34.2 (for python 3.8) via background process having PID 3704 [INFO periodic_update:191]

259 triggered periodic upgrade of pip==20.1.1 (for python 3.8) via background process having PID 3703 [INFO periodic_update:191]

260 got embed update of distribution pip from /home/travis/.local/share/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:133]

261 got embed update of distribution wheel from /home/travis/.local/share/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:133]

270 install setuptools from wheel /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/setuptools-49.2.0-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]

270 Attempting to acquire lock 140630075009200 on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.2.0-py3-none-any.lock [DEBUG filelock:270]

270 Lock 140630075009200 acquired on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.2.0-py3-none-any.lock [INFO filelock:274]

271 build install image for setuptools-49.2.0-py3-none-any.whl to /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.2.0-py3-none-any [DEBUG base:52]

283 install pip from wheel /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-20.1.1-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]

294 Attempting to acquire lock 140629997345616 on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [DEBUG filelock:270]

295 Lock 140629997345616 acquired on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [INFO filelock:274]

296 build install image for pip-20.1.1-py2.py3-none-any.whl to /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any [DEBUG base:52]

297 install wheel from wheel /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]

297 Attempting to acquire lock 140629997343216 on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [DEBUG filelock:270]

297 Lock 140629997343216 acquired on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [INFO filelock:274]

298 build install image for wheel-0.34.2-py2.py3-none-any.whl to /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any [DEBUG base:52]

350 Attempting to release lock 140629997343216 on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [DEBUG filelock:315]

350 Lock 140629997343216 released on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any.lock [INFO filelock:318]

350 copy directory /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:52]

371 copy /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.virtualenv to /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/wheel-0.34.2.virtualenv [DEBUG _sync:52]

372 copy directory /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/wheel [DEBUG _sync:52]

381 generated console scripts wheel wheel-3.8 wheel3.8 wheel3 [DEBUG base:48]

445 Attempting to release lock 140630075009200 on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.2.0-py3-none-any.lock [DEBUG filelock:315]

447 Lock 140630075009200 released on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.2.0-py3-none-any.lock [INFO filelock:318]

448 copy directory /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.2.0-py3-none-any/setuptools-49.2.0.dist-info to /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/setuptools-49.2.0.dist-info [DEBUG _sync:52]

452 copy /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.2.0-py3-none-any/setuptools-49.2.0.virtualenv to /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/setuptools-49.2.0.virtualenv [DEBUG _sync:52]

453 copy /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.2.0-py3-none-any/easy_install.py to /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/easy_install.py [DEBUG _sync:52]

454 copy directory /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.2.0-py3-none-any/setuptools to /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/setuptools [DEBUG _sync:52]

555 copy directory /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.2.0-py3-none-any/pkg_resources to /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/pkg_resources [DEBUG _sync:52]

559 generated console scripts easy_install easy_install-3.8 easy_install3.8 easy_install3 [DEBUG base:48]

749 Attempting to release lock 140629997345616 on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [DEBUG filelock:315]

749 Lock 140629997345616 released on /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any.lock [INFO filelock:318]

751 copy directory /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip-20.1.1.dist-info to /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/pip-20.1.1.dist-info [DEBUG _sync:52]

753 copy directory /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip to /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/pip [DEBUG _sync:52]

860 copy /home/travis/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.1.1-py2.py3-none-any/pip-20.1.1.virtualenv to /home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.8/site-packages/pip-20.1.1.virtualenv [DEBUG _sync:52]

862 generated console scripts pip-3.8 pip pip3 pip3.8 [DEBUG base:48]

862 add activators for Bash, CShell, Fish, PowerShell, Python, Xonsh [INFO session:64]

864 write /home/travis/build/dHannasch/tox-sitepackages-example/py38/pyvenv.cfg [DEBUG pyenv_cfg:34]

865 	home = /opt/python/3.8.0 [DEBUG pyenv_cfg:38]

865 	implementation = CPython [DEBUG pyenv_cfg:38]

865 	version_info = 3.8.0.final.0 [DEBUG pyenv_cfg:38]

865 	virtualenv = 20.0.27 [DEBUG pyenv_cfg:38]

865 	include-system-site-packages = true [DEBUG pyenv_cfg:38]

865 	base-prefix = /opt/python/3.8.0 [DEBUG pyenv_cfg:38]

866 	base-exec-prefix = /opt/python/3.8.0 [DEBUG pyenv_cfg:38]

866 	base-executable = /opt/python/3.8.0/bin/python3.8 [DEBUG pyenv_cfg:38]

866 created virtual environment CPython3.8.0.final.0-64 in 795ms

  creator CPython3Posix(dest=/home/travis/build/dHannasch/tox-sitepackages-example/py38, clear=False, global=True)

  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/travis/.local/share/virtualenv)

    added seed packages: pip==20.1.1, setuptools==49.2.0, wheel==0.34.2

  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator [WARNING __main__:18]

The command "python -m virtualenv -vvv --with-traceback --system-site-packages --no-download py38" exited with 0.

7.66s$ py38/bin/python -m pip install .

Processing /home/travis/build/dHannasch/tox-sitepackages-example

  Installing build dependencies ... -\/done

  Getting requirements to build wheel ... done

    Preparing wheel metadata ... done

ERROR: Could not find a version that satisfies the requirement detectron2 (from tox-sitepackages-example==0.0.1.dev13) (from versions: none)

ERROR: No matching distribution found for detectron2 (from tox-sitepackages-example==0.0.1.dev13)

The command "py38/bin/python -m pip install ." exited with 1.

(This was previously reported at https://github.com/tox-dev/tox/issues/1646 since the ultimate use-case is for tox.)

Aug 09 '20 16:08 dHannasch

Can you replicate this with a docker image? Would speed up finding out what happened significantly.

Aug 09 '20 17:08 gaborbernat

Can you replicate this with a docker image? Would speed up finding out what happened significantly.

Certainly.

dahanna/virtualenv-system-site-packages:pip-install-tox has detectron2 installed (though it doesn't have to be detectron2, you can see the same thing by installing a non-default version of any package) and a pre-created virtualenv called py38 if you want to use that. https://github.com/dHannasch/system-site-packages-dockerfile/blob/pip-install-tox/Dockerfile shows exactly how it's built. You can run e.g. py38/bin/python -m pip install ., or let tox do it for you.

dahanna/virtualenv-system-site-packages:old-tox-no-crash uses an old version of virtualenv instead of installing anew with pip install tox. That image doesn't have the problem, so I presume the problem is caused by an update to some dependency of virtualenv, but I don't know which one (rolling back to an old version of virtualenv itself doesn't help). https://github.com/dHannasch/system-site-packages-dockerfile/blob/old-tox-no-crash/Dockerfile

Aug 10 '20 01:08 dHannasch

What's this FROM dahanna/python:3.8-detectron2?

docker pull dahanna/python:3.8-detectron2                                                                                                                                                                     
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Aug 10 '20 07:08 gaborbernat

I've been trying to make a set of images with some common Python packages pre-installed that anyone can quickly pull and use. (Still sorting out how it should all work, it's tricky to come up with a tagging scheme for multiple packages, which is presumably why no one has done this before. Right now it's based on a handwavy notion of "logical progressions", e.g. numpy->scipy. That tag is actually a typo, that image has Python 3.7. I've been putting off cleaning up the tags since I figured changing the tags while in the middle of debugging would just confuse myself further.) The detectron2 base image isn't really relevant for our purposes, since it's just an image that has detectron2 but not tox installed, is the only point of using it.

https://hub.docker.com/r/dahanna/python/tags https://github.com/dHannasch/alpine-pandas/blob/3.8-detectron2/Dockerfile That's deeply weird that you can't pull it.

Wait, can you pull dahanna/virtualenv-system-site-packages:pip-install-tox?

Aug 10 '20 13:08 dHannasch

I think I've replicated this.

FROM python:3.8-alpine

RUN apk add --no-cache python3 py3-cryptography py3-pip py3-virtualenv

RUN python3 -m venv --system-site-packages venv
RUN venv/bin/pip install --no-cache-dir authlib

Is broken.

FROM alpine:3.13

RUN apk add --no-cache python3 py3-cryptography py3-pip py3-virtualenv

RUN python3 -m venv --system-site-packages venv
RUN venv/bin/pip install --no-cache-dir authlib

Works as expected. Looks like it's a problem with the python:3.8-alpine docker image.

Authlib depends on py3 cryptography.

Apr 04 '21 17:04 traverseda

This request didn't see much support for 2 years, so closing it. If someone still runs into this or wants to fix it we can reopen.

Jun 27 '23 03:06 gaborbernat

virtualenv virtualenv copied to clipboard

--system-site-packages is ignored

virtualenv
virtualenv copied to clipboard