`virtualenv.pyz` invoked via relative path in the fs root under CPython 3.9 seems broken

[webknjaz]

Issue

While having a private message exchange with @gaborbernat earlier today, I hit a weird behavior when attempting to use virtualenv.pyz. We were talking about using it on Fedora and Bernát suggested me to follow https://youtu.be/OXmYKh0eTQ8?t=256 (by @asottile) to try this out.

So I fired up a Fedora 33 container and got an unexpected traceback. It's not fully clear where the bug originates (although I'm leaning towards pointing at CPython) but nevertheless, I wanted to publicly document my findings.

TL;DR It seems like invoking python3.9 virtualenv.pyz (without a leading /) while being in / (and having the zipapp downloaded there) explodes unless one has pre-seeded the cache by invoking it from a different place or by an absolute or explicit relative path.

Environment

Provide at least:

• OS: seems to be irrelevant, but I tried Fedora 33 and the latest ubuntu container images
• It's happening specifically with https://bootstrap.pypa.io/virtualenv.pyz
• Python: 3.9 (while 3.8 seems to be fine)

STR

These commands explode:

$ podman run --rm -it fedora:33 bash -c 'curl -o virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz && pwd && python3.9 virtualenv.pyz venv'
$ podman run --rm -it ubuntu bash -c 'apt update && apt install -y curl python3.9 python3-distutils && curl -o virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz && pwd && python3.9 virtualenv.pyz venv'

And these are fine:

$ podman run --rm -it fedora:33 bash -c 'curl -o virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz && python3.9 /virtualenv.pyz venv'
$ podman run --rm -it fedora:33 bash -c 'curl -o virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz && cd ~ && python3.9 ../virtualenv.pyz venv'
$ podman run --rm -it fedora:33 bash -c 'curl -o virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz && cd ~ && python3.9 /virtualenv.pyz venv && cd / && python3.9 virtualenv.pyz venv'
$ podman run --rm -it ubuntu bash -c 'apt update && apt install -y curl python3-distutils && curl -o virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz && pwd && python3.8 virtualenv.pyz venv'
$ podman run --rm -it fedora:33 bash -c 'dnf install -y python3.8 && curl -o virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz && python3.8 virtualenv.pyz venv'

(replace s/podman/docker/ if that's your poison)

The traceback

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "//virtualenv.pyz/__main__.py", line 168, in <module>
  File "//virtualenv.pyz/__main__.py", line 164, in run
  File "//virtualenv.pyz/virtualenv/__main__.py", line 18, in run
  File "//virtualenv.pyz/virtualenv/run/__init__.py", line 32, in cli_run
  File "//virtualenv.pyz/virtualenv/run/session.py", line 46, in run
  File "//virtualenv.pyz/virtualenv/run/session.py", line 53, in _create
  File "//virtualenv.pyz/virtualenv/create/creator.py", line 171, in run
  File "//virtualenv.pyz/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 101, in create
  File "//virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 89, in create
  File "//virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 92, in install_patch
  File "//virtualenv.pyz/virtualenv/create/via_global_ref/builtin/cpython/cpython3.py", line 27, in env_patch_text
  File "//virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 103, in env_patch_text
  File "/usr/lib64/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "//virtualenv.pyz/virtualenv/app_data/base.py", line 58, in ensure_extracted
  File "/usr/lib64/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "//virtualenv.pyz/virtualenv/app_data/via_disk_folder.py", line 83, in extract
  File "//virtualenv.pyz/virtualenv/util/zipapp.py", line 22, in extract
    
  File "/usr/lib64/python3.9/zipfile.py", line 1429, in getinfo
    raise KeyError(
KeyError: "There is no item named '/virtualenv/create/via_global_ref/_virtualenv.py' in the archive"

[asottile]

can confirm, this seems broken on macos as well

[asottile]

for me it's not always the filesystem root -- I was using /tmp on macos when I tried this at work

[gaborbernat]

This likely seems a bug in the importlib libraries from the way I understand it.

[asottile]

here's a reproducer using docker:

$ docker run --rm -ti ubuntu:jammy bash -exc 'apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get install -qq -y --no-install-recommends curl ca-certificates python3 >& /dev/null && curl --silent -o virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz && python3 virtualenv.pyz venv'
+ apt-get -qq update
+ DEBIAN_FRONTEND=noninteractive
+ apt-get install -qq -y --no-install-recommends curl ca-certificates python3
+ curl --silent -o virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz
+ python3 virtualenv.pyz venv
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "//virtualenv.pyz/__main__.py", line 167, in <module>
  File "//virtualenv.pyz/__main__.py", line 163, in run
  File "//virtualenv.pyz/virtualenv/__main__.py", line 18, in run
  File "//virtualenv.pyz/virtualenv/run/__init__.py", line 32, in cli_run
  File "//virtualenv.pyz/virtualenv/run/session.py", line 46, in run
  File "//virtualenv.pyz/virtualenv/run/session.py", line 53, in _create
  File "//virtualenv.pyz/virtualenv/create/creator.py", line 171, in run
  File "//virtualenv.pyz/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 101, in create
  File "//virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 89, in create
  File "//virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 92, in install_patch
  File "//virtualenv.pyz/virtualenv/create/via_global_ref/builtin/cpython/cpython3.py", line 27, in env_patch_text
  File "//virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 103, in env_patch_text
  File "/usr/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "//virtualenv.pyz/virtualenv/app_data/base.py", line 58, in ensure_extracted
  File "/usr/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "//virtualenv.pyz/virtualenv/app_data/via_disk_folder.py", line 83, in extract
  File "//virtualenv.pyz/virtualenv/util/zipapp.py", line 22, in extract
    
  File "/usr/lib/python3.9/zipfile.py", line 1429, in getinfo
    raise KeyError(
KeyError: "There is no item named '/virtualenv/create/via_global_ref/_virtualenv.py' in the archive"

[gaborbernat]

A PR would be welcomed 👍 (though I suspect the problem will be upstream)

[gaborbernat]

As no one opened upstream issue, closing.