rez-pip
rez-pip copied to clipboard
JeanChristopheMorinPerso
Extras not translated to rez
When running a rez-pip2 on the boxsdk PyPi package with the extra "jwt" specified, the extra dependencies are not included in the resulting package.py file.
Environment
- Windows 10
- Rez version 2.112.0
- Rez python version 3.7
- rez-pip 0.4.0.dev1
To Reproduce
- Install new rez-pip
-
rez-pip2 boxsdk[jwt]==3.9.2 --python-version 3.7
Expected behavior
"jwt" and "cryptography" packages listed in "extra_requires" on boxsdk's setup.py file are included in requires portion of resulting package.py.
Actual behavior
"jwt" and "cryptography" packages are missing from the resulting requires portion of the package.py.
Hi @brycegbrazen , can you provide me with the full output logs please? Also, could you try with rez-pip2 'boxsdk[jwt]==3.9.2' --python-version 3.7 (quoting boxsdk[jwt]==3.9.2)? I remember seeing something like this while I was working on the early PoC...
@JeanChristopheMorinPerso Does it actually log to a file? Not seeing anything in the docs that mention that so I figured I'd ask. Otherwise I can just upload the output from my stdout.
Looks like it includes those extra dependencies, but just doesn't include them as requirements in the package.py. I'll try quoting the package request next.
(Rez) λ rez-pip2 boxsdk[jwt]==3.9.2 --python-version 3.7
INFO Installing requested packages for Python 3.7.9
INFO Resolved 17 dependencies for python 3.7.9
INFO Downloading...
INFO Downloaded PyJWT-2.8.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\PyJWT-2.8.0-py3-none-any.whl' (22591
bytes)
INFO Downloaded idna-3.4 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\idna-3.4-py3-none-any.whl' (61538 bytes)
INFO Downloaded boxsdk-3.9.2 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\boxsdk-3.9.2-py2.py3-none-any.whl'
(139209 bytes)
INFO Downloaded typing_extensions-4.7.1 to
'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\typing_extensions-4.7.1-py3-none-any.whl' (33232 bytes)
INFO Downloaded requests-2.31.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\requests-2.31.0-py3-none-any.whl'
(62574 bytes)
INFO Downloaded urllib3-2.0.7 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\urllib3-2.0.7-py3-none-any.whl'
(124213 bytes)
INFO Downloaded python-dateutil-2.8.2 to
'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\python_dateutil-2.8.2-py2.py3-none-any.whl' (247702 bytes)
INFO Downloaded charset-normalizer-3.3.2 to
'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\charset_normalizer-3.3.2-cp37-cp37m-win_amd64.whl' (98127 bytes)
INFO Downloaded attrs-23.1.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\attrs-23.1.0-py3-none-any.whl' (61160
bytes)
INFO Downloaded requests-toolbelt-1.0.0 to
'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\requests_toolbelt-1.0.0-py2.py3-none-any.whl' (54481 bytes)
INFO Downloaded six-1.16.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\six-1.16.0-py2.py3-none-any.whl' (11053
bytes)
INFO Downloaded importlib-metadata-6.7.0 to
'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\importlib_metadata-6.7.0-py3-none-any.whl' (22934 bytes)
INFO Downloaded zipp-3.15.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\zipp-3.15.0-py3-none-any.whl' (6758
bytes)
INFO Downloaded certifi-2023.7.22 to
'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\certifi-2023.7.22-py3-none-any.whl' (158334 bytes)
INFO Downloaded cffi-1.15.1 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\cffi-1.15.1-cp37-cp37m-win_amd64.whl'
(179301 bytes)
INFO Downloaded pycparser-2.21 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\pycparser-2.21-py2.py3-none-any.whl'
(118697 bytes)
INFO Downloaded cryptography-41.0.5 to
'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\cryptography-41.0.5-cp37-abi3-win_amd64.whl' (2669522 bytes)
INFO Downloaded 17 wheels
INFO Installing boxsdk-3.9.2 wheel
INFO Installing attrs-23.1.0 wheel
INFO Installing cryptography-41.0.5 wheel
INFO Installing PyJWT-2.8.0 wheel
INFO Installing requests-2.31.0 wheel
INFO Installing requests-toolbelt-1.0.0 wheel
INFO Installing urllib3-2.0.7 wheel
INFO Installing python-dateutil-2.8.2 wheel
INFO Installing certifi-2023.7.22 wheel
INFO Installing cffi-1.15.1 wheel
INFO Installing charset-normalizer-3.3.2 wheel
INFO Installing idna-3.4 wheel
INFO Installing six-1.16.0 wheel
INFO Installing importlib-metadata-6.7.0 wheel
INFO Installing typing_extensions-4.7.1 wheel
INFO Installing zipp-3.15.0 wheel
INFO Installing pycparser-2.21 wheel
INFO Creating rez package for boxsdk
INFO Installing boxsdk-3.9.2 []
INFO Created 1 variants and skipped 0
INFO Creating rez package for attrs
INFO Installing attrs-23.1.0 [importlib_metadata]
INFO Created 1 variants and skipped 0
INFO Creating rez package for cryptography
INFO Installing cryptography-41.0.5 [platform-windows, arch-AMD64, python-3.7]
INFO Created 1 variants and skipped 0
INFO Creating rez package for PyJWT
INFO Installing PyJWT-2.8.0 [typing_extensions]
INFO Created 1 variants and skipped 0
INFO Creating rez package for requests
INFO Installing requests-2.31.0 []
INFO Created 1 variants and skipped 0
INFO Creating rez package for requests-toolbelt
INFO Installing requests_toolbelt-1.0.0 []
INFO Created 1 variants and skipped 0
INFO Creating rez package for urllib3
INFO Installing urllib3-2.0.7 []
INFO Created 1 variants and skipped 0
INFO Creating rez package for python-dateutil
INFO Installing python_dateutil-2.8.2 []
INFO Created 1 variants and skipped 0
INFO Creating rez package for certifi
INFO Installing certifi-2023.7.22 []
INFO Created 1 variants and skipped 0
INFO Creating rez package for cffi
INFO Installing cffi-1.15.1 [platform-windows, arch-AMD64, python-3.7]
INFO Created 1 variants and skipped 0
INFO Creating rez package for charset-normalizer
INFO Installing charset_normalizer-3.3.2 [platform-windows, arch-AMD64, python-3.7]
INFO Created 1 variants and skipped 0
INFO Creating rez package for idna
INFO Installing idna-3.4 []
INFO Created 1 variants and skipped 0
INFO Creating rez package for six
INFO Installing six-1.16.0 []
INFO Created 1 variants and skipped 0
INFO Creating rez package for importlib-metadata
INFO Installing importlib_metadata-6.7.0 [typing_extensions-3.6.4+]
INFO Created 1 variants and skipped 0
INFO Creating rez package for typing_extensions
INFO Installing typing_extensions-4.7.1 []
INFO Created 1 variants and skipped 0
INFO Creating rez package for zipp
INFO Installing zipp-3.15.0 []
INFO Created 1 variants and skipped 0
INFO Creating rez package for pycparser
INFO Installing pycparser-2.21 []
INFO Created 1 variants and skipped 0
I just tried on my side and it's working. And from what I see in your log, it seems like it also worked for you. The extra is called jwt and it installs PyJWT and cryptography (https://github.com/box/box-python-sdk/blob/v3.9.2/setup.py#L63)
I see
INFO Creating rez package for cryptography
INFO Installing cryptography-41.0.5 [platform-windows, arch-AMD64, python-3.7]
INFO Created 1 variants and skipped 0
INFO Creating rez package for PyJWT
INFO Installing PyJWT-2.8.0 [typing_extensions]
INFO Created 1 variants and skipped 0
Which means it created a package cryptography-41.0.5 and PyJWT-2.8.0.
Yeah I see it's installing those two extra dependencies, but it doesn't actually include them in the resolve when I run a
rez-env boxsdk. I thought it'd include PyJWT and cryptography as dependencies in the package.py somehow.
Ah yeah. Well, the boxsdk package will be boxsdk only... We don't try to merge things (as it would be had to know what you really want to do. If you pip install boxsdk[jwt], you will endup with everything on one env, but it doesn't modify dependencies. In other words
$ python
>>> import importlib.metadata
>>> dist = importlib.metadata.distribution('boxsdk')
>>> dist.requires
['attrs (>=17.3.0)', 'urllib3', 'requests (<3,>=2.4.3)', 'requests-toolbelt (>=0.4.0)', 'python-dateutil', "coveralls ; extra == 'coveralls'", "tox (<=3.28.0) ; extra == 'coveralls'", "tox (<=3.28.0) ; extra == 'dev'", "tox-gh-actions ; extra == 'gh'", "tox (<=3.28.0) ; extra == 'gh'", "pyjwt (>=1.7.0) ; extra == 'jwt'", "cryptography (>=3) ; extra == 'jwt'", "redis (>=2.10.3) ; extra == 'redis'", "bottle ; extra == 'test'", "jsonpatch (>1.14) ; extra == 'test'", "sqlalchemy (<1.4.0) ; extra == 'test'", "pytest ; extra == 'test'", "pytest-timeout ; extra == 'test'", "pytest-cov ; extra == 'test'", "pytest-lazy-fixture ; extra == 'test'", "pytz ; extra == 'test'", "urllib3 (<2) ; extra == 'test'"]
The dependencies aren't merged. The xtras are still extras even after the install.
rez-pip does the same (both the builtin rez-pip and rez-pip2).
We additionally don't have a good way to translate boxsdk[jwt] to rez-env boxsdk[jwt]. See https://github.com/JeanChristopheMorinPerso/rez-pip/blob/0217569e14dff414ad3372dd3909536944a6c7c9/src/rez_pip/utils.py#L490-L493. We simply don't have a way to do that right now.
Maybe ephemerals could help, but it would need some investigation to see if they would work or not.