The virtualenv.pyz zipapp cannot be run with an arbitrary python.

[jsirois]

Issue

The virtualenv.pyz zipapp cannot be run with an arbitrary python if that python already has an incompatible version of virtualenv installed. This makes scripting virtualenv hard.

Environment

• OS: Linux
• pip list of the host python where virtualenv is installed: N/A - see Reproduction steps below.

Reproduction Steps

• All good baseline:

$ docker run --rm -it ubuntu:20.04
root@789c91e7e2d3:/# apt update &>/dev/null && apt install -y curl python3 python3-distutils &>/dev/null
root@789c91e7e2d3:/# curl -sSL -O https://raw.githubusercontent.com/pypa/get-virtualenv/20.4.7/public/virtualenv.pyz
root@789c91e7e2d3:/# python3 virtualenv.pyz a.venv
created virtual environment CPython3.8.5.final.0-64 in 274ms
  creator CPython3Posix(dest=/a.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

• Blow things up by installing virtualenv:

root@789c91e7e2d3:/# apt install -y python3-virtualenv &>/dev/null
root@789c91e7e2d3:/# python3 virtualenv.pyz b.venv
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "virtualenv.pyz/__main__.py", line 168, in <module>
  File "virtualenv.pyz/__main__.py", line 164, in run
  File "virtualenv.pyz/virtualenv/__main__.py", line 18, in run
  File "virtualenv.pyz/virtualenv/run/__init__.py", line 30, in cli_run
  File "virtualenv.pyz/virtualenv/run/__init__.py", line 48, in session_via_cli
  File "virtualenv.pyz/virtualenv/run/__init__.py", line 75, in build_parser
  File "virtualenv.pyz/virtualenv/run/plugin/seeders.py", line 8, in __init__
  File "virtualenv.pyz/virtualenv/run/plugin/base.py", line 39, in options
  File "virtualenv.pyz/virtualenv/run/plugin/base.py", line 18, in entry_points_for
  File "virtualenv.pyz/virtualenv/run/plugin/base.py", line 18, in <genexpr>
  File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'

This is the same end problem as #1873 but its a bit less under user control. They've downloaded the zipapp, run python virtualenv.pyz and got bitten by the fact that python already had virtualenv installed.

Over in Pants where we used to use the virtualenv.pyz to bootstrap a virtualenv for Pants we had to switch to using Pex to run virtualenv since Pex can create a zipapp with proper isolation from the underlying interpreter used to run the PEX zipapp: https://github.com/pantsbuild/setup/pull/99

[jsirois]

If you all would be interested to moving to PEX packaging instead of or in addition to your current .pyz packaging; I'd be happy to work on a PR to add that support.

[gaborbernat]

If you all would be interested to moving to PEX packaging instead of or in addition to your current .pyz packaging; I'd be happy to work on a PR to add that support.

PEX is not a good solution for our use case. It supports Windows at a best-effort level only, and more importantly, it is able to use only a single dependency version. Note our dependency for Python 2.7 and 3.4 are totally different than 3.9 (mostly because for some dependencies there's no one dependency version that supports all target Python versions from 2.7 to 3.10). When running the zipapp with 3.9 we want to use the latest 3.9 compatible dependencies, while on 2.7 the same for 2.7.

The problem you're facing here can be solved with a simple sys.path manipulation within the __main__.py of the zipapp, and a PR doing that would be welcomed.

[jsirois]

Ah, yes Windows (and Python 3.4). I'll take a crack at a PR to fixup the existing zipapp main module as you suggest.

I do want to note that Pex does support all the rest though. For example (I use --include-tools and PEX_TOOLS=1 here just to have a concise way to demonstrate that multiple versions of distributions are contained in the PEX file and the right ones are activated for the respective interpreter):

$ pex --interpreter-constraint ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" virtualenv -c virtualenv -o virtualenv.pex --include-tools
$ PEX_TOOLS=1 ./virtualenv.pex info -i4
{
    "always_write_cache": false, 
    "build_properties": {
        "class": "PyPy", 
        "pex_version": "2.1.42", 
        "platform": "manylinux_2_33_x86_64", 
        "version": [
            2, 
            7, 
            18
        ]
    }, 
    "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709", 
    "distributions": {
        "appdirs-1.4.4-py2.py3-none-any.whl": "8b8a15c0d198ccfae8ee432324665d24aeef25e3", 
        "configparser-4.0.2-py2.py3-none-any.whl": "6fdfab75419831b52e6da0d0b0f50fd861cce6fe", 
        "contextlib2-0.6.0.post1-py2.py3-none-any.whl": "7798441a9367996aae652baa225494ccd01f74d2", 
        "distlib-0.3.2-py2.py3-none-any.whl": "3f30172a0b5bce395d4b7dc77cbbd3a219745649", 
        "filelock-3.0.12-py2-none-any.whl": "96e0dee3b9370e1c2648118637217a0c76e1336c", 
        "filelock-3.0.12-py3-none-any.whl": "03d3bd874fa4af18d755d7f2717e3cc9f2f53814", 
        "importlib_metadata-2.1.1-py2.py3-none-any.whl": "eaf8882c291b6c0e51569ae0f7c8c4d0d52cca26", 
        "importlib_metadata-4.5.0-py3-none-any.whl": "301eb5682391c68c48cdbcdf3b04402aadbc5ad0", 
        "importlib_resources-3.2.1-py2.py3-none-any.whl": "d0f724718a374055b6fa092ec475edf53d2381cb", 
        "importlib_resources-3.3.1-py2.py3-none-any.whl": "961a0d57dc2693464538ba13c36c2dffe38e2ca6", 
        "importlib_resources-5.1.4-py3-none-any.whl": "85e09503e1b7d112114c2388063440288e102450", 
        "pathlib2-2.3.5-py2.py3-none-any.whl": "9bfaf31a2df7f99d43d97976de1c2c4414fb9639", 
        "scandir-1.10.0-cp27-cp27mu-linux_x86_64.whl": "e716d31f3544810846e6d32240c22488a0e87467", 
        "scandir-1.10.0-pp27-pypy_73-linux_x86_64.whl": "674c98d613503fc00429bdc583d313224eb8786e", 
        "singledispatch-3.6.2-py2.py3-none-any.whl": "9f516b43610a409bb73b0b756a002d7fd8fafeca", 
        "six-1.16.0-py2.py3-none-any.whl": "035d7c208925c1832def39b592f3477ca36397bf", 
        "typing-3.10.0.0-py2-none-any.whl": "316e046b1c00ceb00c35385c0aca6a874347d671", 
        "typing_extensions-3.10.0.0-py3-none-any.whl": "2d21ea9841868bec222ae7bffdf3212ac3ad9c3e", 
        "virtualenv-20.4.7-py2.py3-none-any.whl": "6a8ad75c3ffc00512cfea6662e102c25fa216cc7", 
        "zipp-1.2.0-py2.py3-none-any.whl": "a90a22bea529301bc39f1de526612414e3cd5dff", 
        "zipp-3.4.1-py3-none-any.whl": "6808f40f165a419dc498ee4e88d5de6d2b9b8ef7"
    }, 
    "emit_warnings": true, 
    "entry_point": "virtualenv.__main__:run_with_catch", 
    "ignore_errors": false, 
    "inherit_path": "false", 
    "interpreter_constraints": [
        ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
    ], 
    "pex_hash": "5961d9d9b4c0146d23c3a7164abb3b84a39d449d", 
    "pex_path": null, 
    "requirements": [
        "virtualenv"
    ], 
    "strip_pex_env": true, 
    "unzip": false, 
    "venv": false, 
    "venv_bin_path": "False", 
    "venv_copies": false, 
    "zip_safe": true, 
    "pex_root": "/home/jsirois/.pex"
}

Under CPython 2.7:

$ PEX_TOOLS=1 python2.7 virtualenv.pex repository info | cut -d' ' -f-2
virtualenv 20.4.7
appdirs 1.4.4
distlib 0.3.2
filelock 3.0.12
six 1.16.0
pathlib2 2.3.5
scandir 1.10.0
importlib-resources 3.3.1
contextlib2 0.6.0.post1
singledispatch 3.6.2
typing 3.10.0.0
zipp 1.2.0
importlib-metadata 2.1.1
configparser 4.0.2

Under PyPy (2.7):

$ PEX_TOOLS=1 pypy virtualenv.pex repository info | cut -d' ' -f-2
virtualenv 20.4.7
appdirs 1.4.4
distlib 0.3.2
filelock 3.0.12
six 1.16.0
pathlib2 2.3.5
scandir 1.10.0
importlib-resources 3.3.1
contextlib2 0.6.0.post1
singledispatch 3.6.2
typing 3.10.0.0
zipp 1.2.0
importlib-metadata 2.1.1
configparser 4.0.2

Note that PyPy gets its own platform-specific wheel activated for scandir.

For CPython 3.5:

$ PEX_TOOLS=1 python3.5 virtualenv.pex repository info | cut -d' ' -f-2
virtualenv 20.4.7
appdirs 1.4.4
distlib 0.3.2
filelock 3.0.12
six 1.16.0
importlib-resources 3.2.1
zipp 1.2.0
importlib-metadata 2.1.1

And for CPython 3.6:

$ PEX_TOOLS=1 python3.6 virtualenv.pex repository info | cut -d' ' -f-2
virtualenv 20.4.7
appdirs 1.4.4
distlib 0.3.2
filelock 3.0.12
six 1.16.0
importlib-resources 5.1.4
zipp 3.4.1
importlib-metadata 4.5.0
typing-extensions 3.10.0.0

Note that different versions of importlib-metadata and importlib-resources are embedded and chosen. Etc.

[gaborbernat]

Guess we can consider moving to it when they make Windows a first-class citizen then.

[jsirois]

Hrm, so the sys.path ordering is just fine. the virtualenv.pyz is in the 1st slot and virtualenv imports from there. That's, of course, what leads to the subsequent error when we try to load() an entrypoint scanned from an older virtualenvs dist-info metadata that's later on the sys.path.

IOW, this hack fixes:

$ git diff src/
diff --git a/src/virtualenv/run/plugin/base.py b/src/virtualenv/run/plugin/base.py
index ed10fe0..1ecf1c7 100644
--- a/src/virtualenv/run/plugin/base.py
+++ b/src/virtualenv/run/plugin/base.py
@@ -1,5 +1,6 @@
 from __future__ import absolute_import, unicode_literals
 
+import logging
 import sys
 from collections import OrderedDict
 
@@ -15,7 +16,15 @@ class PluginLoader(object):
 
     @classmethod
     def entry_points_for(cls, key):
-        return OrderedDict((e.name, e.load()) for e in cls.entry_points().get(key, {}))
+        loadable_entry_points = OrderedDict()
+        for ep in cls.entry_points().get(key, {}):
+            try:
+                loadable_entry_points[ep.name] = ep.load()
+            except ModuleNotFoundError as err:
+                if not ep.module.startswith("virtualenv."):
+                    raise
+                logging.warning("Failed to load {}: {} {}".format(ep, err, __file__))
+        return loadable_entry_points
 
     @staticmethod
     def entry_points():

Ideally, the EntryPoint object would carry provenance and we'd just skip any entrypoint that came from a virtualenv distribution other than the current one. Less ideally, this hack could check the additional condition that virtualenv was loaded from a zipapp.

@gaborbernat were you seeing something I'm not here? Maybe some fancy hooking along these lines?: https://importlib-resources.readthedocs.io/en/latest/using.html#extending that tries to take over all resource discovery both inside the pyz sys.path entry and across all the other sys.payh entries too? That sort of thing could de-dup resources favoring the 1st sys.path entry that has a given root module / package (like virtualenv).

[gaborbernat]

Any reason why we can't just remove the sys path entry belonging to the current python intrerpreters platlib and purelib? So that whatever is in those locations never gets discovered?

[jsirois]

Is that legit? I'll try that if its ok. I assumed your plugin mechanism was meant to allow plugins on the sys.path and this would defeat that IIUC.

[gaborbernat]

I don't think we should support plugins for zipapps. If someone wants to use a plugin with zipapp they likely want to repackage the zipapp with the plugin included 🤔

[jsirois]

Ok. Sounds good to me. I'll pick this back up with that in mind.

[mayeut]

A simple workaround that seems to work is to invoke python with the -S flag ("don't imply 'import site' on initialization")

[gaborbernat]

Closing as above.