Shiv does not fully package namespaced packages

[stephenhelms]

Hi, I've run into a corner case when trying to package an application that uses tensorflow with shiv. You can reproduce the issue by creating a python 3.6 virtual environment with tensorflow-cpu==2.6.0, then running: shiv -c tensorboard -o brokenshiv tensorboard

Tensorflow uses a few namespaced packages from google, which end up in a few locations:

• lib/python3.6/site-packages/google/auth
• lib/python3.6/site-packages/google/oauth
• lib64/python3.6/site-packages/google/protobuf

When shiv runs its packaging, it executes pip install --target, which triggers this linked issue: https://github.com/pypa/pip/issues/10110 During the installation, there is a warning: WARNING: Target directory /tmp/tmpg5wic5v5/google already exists. Specify --upgrade to force replacement.

In the unzipped shiv package, only the auth and oauth directories make it into site-packages under google; the protobuf package is missing.

In the pip issue, the maintainers suggest the correct behavior is to use --prefix during installation. Do you have a tip for how to solve this? Thanks!

[stephenhelms]

Also this other issue looks like it is probably the same problem: https://github.com/linkedin/shiv/issues/187

[lorencarvalho]

Hi @stephenhelms,

I can't reproduce this issue:

darwin ~ $ shiv -c tensorboard -o brokenshiv tensorboard
Collecting tensorboard
  Downloading tensorboard-2.6.0-py3-none-any.whl (5.6 MB)
Collecting requests<3,>=2.21.0
  Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting markdown>=2.6.8
  Downloading Markdown-3.3.4-py3-none-any.whl (97 kB)
Collecting tensorboard-plugin-wit>=1.6.0
  Downloading tensorboard_plugin_wit-1.8.0-py3-none-any.whl (781 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting setuptools>=41.0.0
  Using cached setuptools-58.2.0-py3-none-any.whl (946 kB)
Collecting werkzeug>=0.11.15
  Downloading Werkzeug-2.0.2-py3-none-any.whl (288 kB)
Collecting grpcio>=1.24.3
  Downloading grpcio-1.41.0-cp38-cp38-macosx_10_10_x86_64.whl (3.9 MB)
Collecting google-auth<2,>=1.6.3
  Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting protobuf>=3.6.0
  Downloading protobuf-3.18.1-cp38-cp38-macosx_10_9_x86_64.whl (1.0 MB)
Collecting tensorboard-data-server<0.7.0,>=0.6.0
  Downloading tensorboard_data_server-0.6.1-py3-none-macosx_10_9_x86_64.whl (3.5 MB)
Collecting numpy>=1.12.0
  Using cached numpy-1.21.2-cp38-cp38-macosx_10_9_x86_64.whl (16.9 MB)
Collecting wheel>=0.26
  Downloading wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Collecting absl-py>=0.4
  Downloading absl_py-0.14.1-py3-none-any.whl (131 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting cachetools<5.0,>=2.0.0
  Downloading cachetools-4.2.4-py3-none-any.whl (10 kB)
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting requests-oauthlib>=0.7.0
  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
  Downloading oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Installing collected packages: urllib3, pyasn1, idna, charset-normalizer, certifi, six, setuptools, rsa, requests, pyasn1-modules, oauthlib, cachetools, requests-oauthlib, google-auth, wheel, werkzeug, tensorboard-plugin-wit, tensorboard-data-server, protobuf, numpy, markdown, grpcio, google-auth-oauthlib, absl-py, tensorboard

shiv appears to successfully create a tensorboard executable for me, without any warnings or errors about the google namespace:

darwin ~ $ ls ~/.shiv/brokenshiv_1ce8427354c8b384181ba0f232b442312ce8dfffe1e1c3dc4ffe521ee5b08926/site-packages/google
auth  oauth2  protobuf

The pip issue that you linked is talking about running multiple invocations of pip install --target, which shiv does not do (it only runs it once, to stage the dependencies prior to packing them into the zip). That said, I don't expect that the included pip behavior will work for 100% of use cases that folks want to use shiv for, which is why it also includes a --site-packages argument, which sidesteps the pip install --target code path in favor of just copying in an existing site-packages directory.

Hope that helps!