virtualenv icon indicating copy to clipboard operation
virtualenv copied to clipboard
verified publisher icon pypa

zipapp virtual environment creation fails if zipapp path is symlinked

Open andyneff opened this issue 5 years ago • 13 comments

Issue

On a fresh new system, trying to use the zipapp to setup a virtualenv when the pyz files is referenced by a symlink, it fails saying KeyError: "There is no item named 'nv/create/via_global_ref/_virtualenv.py' in the archive"

But it works on my machine?

Yes, if you ever used the virtualenv (zipapp?) successfully, it creates files in ~/Library/Application Support/virtualenv (on macOS) or ~/.local/share/virtualenv/ (on Linux) and the existence of these files (unzip and wheel) makes the zipapp work. By deleting that directory, you should be able to reproduce the problem. This was first discovered on travis-ci, where the machine is new every time.

Environment

  • OS: macOS 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64.
  • Python 3.7.7. (from brew), also tried 3.7.9

Also reproduced on

  • OS: fedora:31
  • Python 3.7.9

Output of the virtual environment creation

bash-3.2$ (cd /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo; python3 /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz -vvv --with-traceback /var/folders/g0/944401t11gz1jl7vxt
nzqs_m0001yj/T/foo/pipenv-2020.6.2)
206 setup logging to NOTSET [DEBUG report:43]
219 find interpreter for spec PythonSpec(path=/usr/local/opt/python/bin/python3.7) [INFO builtin:44]
219 proposed PythonInfo(spec=CPython3.7.7.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.7 (default, Mar 10 2020, 15:43:33) \n[Clang 11.0.0 (clang-1100.0.33.17)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:50]
219 accepted PythonInfo(spec=CPython3.7.7.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.7 (default, Mar 10 2020, 15:43:33) \n[Clang 11.0.0 (clang-1100.0.33.17)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:52]
224 filesystem is not case-sensitive [DEBUG info:28]
263 create virtual environment via CPython3Posix(dest=/private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2, clear=False, global=False) [INFO session:52]
263 write /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/pyvenv.cfg [DEBUG pyenv_cfg:34]
263     home = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
263     implementation = CPython [DEBUG pyenv_cfg:38]
263     version_info = 3.7.7.final.0 [DEBUG pyenv_cfg:38]
263     virtualenv = 20.0.31 [DEBUG pyenv_cfg:38]
263     include-system-site-packages = false [DEBUG pyenv_cfg:38]
264     base-prefix = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
264     base-exec-prefix = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
264     base-executable = /usr/local/opt/python/bin/python3.7 [DEBUG pyenv_cfg:38]
264 remove file /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/bin/python [DEBUG _sync:38]
264 symlink /usr/local/opt/python/bin/python3.7 to /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/bin/python [DEBUG _sync:44]
265 Attempting to acquire lock 4396469136 on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [DEBUG filelock:270]
265 Lock 4396469136 acquired on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [INFO filelock:274]
265 extract /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/_virtualenv.py to /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py [DEBUG zipapp:19]
267 Attempting to release lock 4396469136 on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [DEBUG filelock:315]
267 Lock 4396469136 released on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [INFO filelock:318]
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/__main__.py", line 168, in <module>
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/__main__.py", line 164, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/__main__.py", line 16, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/__init__.py", line 28, in cli_run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/session.py", line 46, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/session.py", line 53, in _create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/creator.py", line 157, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 86, in create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 88, in create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 91, in install_patch
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/cpython/cpython3.py", line 27, in env_patch_text
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 102, in env_patch_text
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/app_data/base.py", line 54, in ensure_extracted
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/app_data/via_disk_folder.py", line 81, in extract
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/util/zipapp.py", line 22, in extract

  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/zipfile.py", line 1431, in getinfo
    'There is no item named %r in the archive' % name)
KeyError: "There is no item named 'nv/create/via_global_ref/_virtualenv.py' in the archive"

Note: Notice /virtuale is missing from the KeyError, the exact length /private which is the difference between the TMPDIR and the real path of the TMPDIR on macOS

On Fedora, the last line becomes:

KeyError: "There is no item named '/virtualenv/create/via_global_ref/_virtualenv.py' in the archive"

Steps to reproduce

  1. docker run -it --rm fedora:31
  2. mkdir -p /tmp/foo/bar
  3. ln -s /tmp/foo/bar /tmp/foo/bar2
  4. curl -L -o /tmp/foo/bar/virtualenv.pyz https://bootstrap.pypa.io/virtualenv/3.7/virtualenv.pyz
  5. python3 /tmp/foo/bar2/virtualenv.pyz /tmp/foo/blah

Here are some one-liners to reproduce the error

docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" /tmp/blah'

docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" /tmp/blah'

Not all values of REAL/SYM fail. It appears if the length of the symlink path and realpath differ, it fails. If they are the exact same length, no error occurs

Fail

REAL SYM
/tmp/foo/bar /tmp/bar/bar2
/tmp/foo/bar /tmp/foo/bc
/tmp/foo/car /tmp/foo/bb
/tmp/bb /tmp/aaa

Does not fail

REAL SYM
/tmp/foo /tmp/bar
/tmp/foo/bar /tmp/foo/abc
/tmp/foo/car /tmp/foo/bbb

andyneff avatar Sep 25 '20 19:09 andyneff

Wouldn't this be fixed if you're using the copy method, instead of the symlink one? The symlink method cannot work if you're deleting the app-data folder from where we symlink. 🤔 ?

gaborbernat avatar Oct 08 '20 05:10 gaborbernat

@gaborbernat Are you saying that a user should have to add flags when calling the zipapp?

docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" --copies /tmp/blah'
docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" --symlink-app-data /tmp/blah'

both fail the same

andyneff avatar Oct 08 '20 14:10 andyneff

I wonder why the copy fails, it should not, but will look into it. If you have free time to do it, and you can do it that would be even better 👍

gaborbernat avatar Oct 08 '20 15:10 gaborbernat

Seems no longer an issue as no one reported it in 3 years.

gaborbernat avatar Jun 27 '23 03:06 gaborbernat

@gaborbernat: please reopen, I have just ran into this issue with virtualenv 20.24.0 on MacOS Catalina. (Why I'm still using Catalina is a long story.)

It seems to me that it's important that the .pyz is referred to through the symlink, i.e. if I cd into the directory, and only pass the name of the .pyz, then it works, but if I use the full path (which contains a symlink), then it fails.

petamas avatar Jul 19 '23 15:07 petamas

Please fill in a PR to fix it, we do not plan to work on this ourselves.

gaborbernat avatar Jul 19 '23 15:07 gaborbernat

Thanks for the quick reply! I have no idea where in the code the problem is, but I'll try to take a look.

(For now, I worked around it by calling os.path.realpath() on the path to the .pyz before calling it in my scripts.)

petamas avatar Jul 19 '23 15:07 petamas

Upon investigation of the code, it seems like addibg os.path.realpath to this line would solve the issue: https://github.com/pypa/virtualenv/blob/328f965f0dea2e4963ca9238d49233bf0f142bd8/src/virtualenv/util/zipapp.py#L26

ROOT has this function called, while full_path does not, so ROOT is not necessarily a prefix of full_path, making sub_file contain something weird. This also explains @andyneff 's observation about path lengths.

I'll try fixing it and PRing the fix later this week or next week.

petamas avatar Jul 19 '23 15:07 petamas

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

jamesnyao avatar Jan 17 '24 00:01 jamesnyao

I'm seeing this issue too when building electron Python 3.11.8+chromium.31

$ '/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/bin/python3' '-B' '-I' '/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz' '--no-download' '--always-copy' '/home/testuser/.cache/.vpython-root/store/python_venv-478cu52ikjb86ilmh6apfpko0k/contents'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/__main__.py", line 167, in <module>
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/__main__.py", line 163, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/__main__.py", line 16, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/run/__init__.py", line 30, in cli_run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/run/session.py", line 42, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/run/session.py", line 49, in _create
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/creator.py", line 153, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 93, in create
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 83, in create
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 86, in install_patch
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/cpython/cpython3.py", line 25, in env_patch_text
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 97, in env_patch_text
  File "/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/app_data/base.py", line 54, in ensure_extracted
  File "/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/app_data/via_disk_folder.py", line 78, in extract
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/util/zipapp.py", line 19, in extract
    import {module}
                   ^
  File "/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/lib/python3.11/zipfile.py", line 1484, in getinfo
    raise KeyError(
KeyError: "There is no item named 'lenv.pyz/virtualenv/create/via_global_ref/_virtualenv.py' in the archive"

bruceauyeung avatar Feb 26 '24 12:02 bruceauyeung

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

did you fix/bypass your problem ?

bruceauyeung avatar Feb 27 '24 01:02 bruceauyeung

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

did you fix/bypass your problem ?

https://github.com/pypa/virtualenv/pull/2613 merged after virtualenv-20.17.1 being released. So backporting those modifications to 20.17.1 can fix the problem. steps:

  1. cd /home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/
  2. mv virtualenv.pyz virtualenv.pyz.bak
  3. unzip virtualenv.pyz.bak -d virtualenv
  4. vi virtualenv/virtualenv/util/zipapp.py

replace contents in zipapp.py

full_path = os.path.abspath(str(full_path))
    sub_file = full_path[len(ROOT) + 1 :]

to

  full_path = os.path.realpath(os.path.abspath(str(full_path)))
  prefix = ROOT + os.sep
  assert full_path.startswith(prefix), f"full_path={full_path} should start with prefix={prefix}"
  sub_file = full_path[len(prefix) :]
  1. python3 -m zipapp virtualenv

bruceauyeung avatar Feb 29 '24 03:02 bruceauyeung

Tried https://github.com/pypa/virtualenv/issues/1949#issuecomment-1970342800 solution, and re-run gclient sync command, the errors are gone.

Thanks

Zack4396 avatar Mar 01 '24 06:03 Zack4396

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

did you fix/bypass your problem ?

#2613 merged after virtualenv-20.17.1 being released. So backporting those modifications to 20.17.1 can fix the problem. steps:

  1. cd /home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/
  2. mv virtualenv.pyz virtualenv.pyz.bak
  3. unzip virtualenv.pyz.bak -d virtualenv
  4. vi virtualenv/virtualenv/util/zipapp.py

replace contents in zipapp.py

full_path = os.path.abspath(str(full_path))
    sub_file = full_path[len(ROOT) + 1 :]

to

  full_path = os.path.realpath(os.path.abspath(str(full_path)))
  prefix = ROOT + os.sep
  assert full_path.startswith(prefix), f"full_path={full_path} should start with prefix={prefix}"
  sub_file = full_path[len(prefix) :]
  1. python3 -m zipapp virtualenv

I just got this error on a fresh download of the chromium dev kit on ubuntu -- this workaround worked for me too but I had to move the virtualenv.pyz.bak file out of the way first... Thank you!!!

josegal avatar Apr 09 '24 16:04 josegal