pipenv
pipenv copied to clipboard
pypa
Version of pipenv > 2020.11.15 will produce error AttributeError: 'JoinedStr' object has no attribute 's'
Issue description
Version of pipenv > 2020.11.15 will produce error AttributeError: 'JoinedStr' object has no attribute 's' when it installs a package from private repo that contains setup.py with dependencies from private repo.
Expected result
Packages and its dependencies should install from private repo (bitbucket)
Actual result
pipenv install --verbose
Installing dependencies from Pipfile.lock (19667c)...
/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py:226: PipDeprecationWarning: DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
issue=7555
Traceback (most recent call last):
File "/home/SYS/comp1/.pyenv/versions/3.7.13/bin/pipenv", line 8, in <module>
sys.exit(cli())
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/cli/command.py", line 215, in install
site_packages=state.site_packages
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/core.py", line 2008, in do_install
keep_outdated=keep_outdated
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/core.py", line 1269, in do_init
pypi_mirror=pypi_mirror,
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/core.py", line 824, in do_install_dependencies
project, normal_deps, procs, failed_deps_queue, requirements_dir, **install_kwargs
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/core.py", line 688, in batch_install
dep for dep in deps_to_install if not project.environment.is_satisfied(dep)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/core.py", line 688, in <listcomp>
dep for dep in deps_to_install if not project.environment.is_satisfied(dep)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/environment.py", line 776, in is_satisfied
vcs_type == req.vcs and commit_id == req.commit_hash
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/cached_property.py", line 36, in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2491, in commit_hash
with self.req.locked_vcs_repo() as repo:
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2223, in locked_vcs_repo
self._parsed_line.vcsrepo = vcsrepo
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 907, in vcsrepo
kwargs=wheel_kwargs,
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 1598, in create
created.get_initial_info()
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 1431, in get_initial_info
parsed.update(self.parse_setup_py())
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 1162, in parse_setup_py
parsed = ast_parse_setup_py(self.setup_py.as_posix())
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 880, in ast_parse_setup_py
return SetupReader.read_setup_py(Path(path), raising)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 205, in read_setup_py
"install_requires": caller(cls._find_install_requires, setup_call, body),
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 197, in caller
return func(*args, **kwargs)
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 360, in _find_install_requires
return [el.s for el in value.elts]
File "/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 360, in <listcomp>
return [el.s for el in value.elts]
AttributeError: 'JoinedStr' object has no attribute 's'
Steps to replicate
Pipfile:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
pylint = "*"
autopep8 = "*"
[packages]
requests = "==2.*"
matplotlib = "*"
custom_lib = {git = '[email protected]/org/custom_lib.git'}
[requires]
python_version = "3.7"
custom_lib -> setup.py:
'''
Custom Lib
'''
from setuptools import setup
HOST = 'git+ssh://[email protected]/org/'
setup(
author='Team',
name='custom_lib',
version='2.0.4',
description='Custom Lib',
install_requires=['easysnmp', f'custom_lib@{HOST}custom_lib.git#egg=custom_lib'],
py_modules=['custom_lib']
)
Steps:
-
pipenv install
pipenv --support
$ pipenv --support
Pipenv version: '2022.1.8'
Pipenv location: '/home/SYS/comp1/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pipenv'
Python location: '/home/SYS/comp1/.pyenv/versions/3.7.13/bin/python3'
Python installations found:
-
3.10.5:/home/SYS/comp1/.pyenv/versions/3.10.5/bin/python3 -
3.10.4:/usr/bin/python3 -
3.10.4:/usr/bin/python -
3.10.4:/bin/python3 -
3.10.4:/bin/python -
3.9.13:/usr/bin/python3.9 -
3.9.13:/bin/python3.9 -
3.7.13:/home/SYS/comp1/.pyenv/versions/3.7.13/bin/python3 -
3.7.13:/home/SYS/comp1/.pyenv/versions/3.7.13/bin/python -
3.7.13:/home/SYS/comp1/.pyenv/versions/3.7.13/bin/python3.7m -
3.7.13:/home/SYS/comp1/.pyenv/versions/3.7.13/bin/python3.7 -
3.7.13:/home/SYS/comp1/.pyenv/versions/3.7.13/bin/python3 -
3.7.9:/home/SYS/comp1/.pyenv/versions/3.7.9/bin/python3 -
3.6.11:/home/SYS/comp1/.pyenv/versions/3.6.11/bin/python3 -
3.6.9:/usr/bin/python3.6m -
3.6.9:/usr/bin/python3.6 -
3.6.9:/bin/python3.6m -
3.6.9:/bin/python3.6 -
3.5.10:/usr/bin/python3.5 -
3.5.10:/usr/bin/python3.5m -
3.5.10:/bin/python3.5 -
3.5.10:/bin/python3.5m -
2.7.18:/usr/bin/python2.7 -
2.7.18:/usr/bin/python2 -
2.7.18:/bin/python2.7 -
2.7.18:/bin/python2
PEP 508 Information:
{'implementation_name': 'cpython',
'implementation_version': '3.7.13',
'os_name': 'posix',
'platform_machine': 'x86_64',
'platform_python_implementation': 'CPython',
'platform_release': '5.15.0-39-generic',
'platform_system': 'Linux',
'platform_version': '#42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022',
'python_full_version': '3.7.13',
'python_version': '3.7',
'sys_platform': 'linux'}
System environment variables:
-
SHELL -
PYENV_HOOK_PATH -
PYENV_SHELL -
NVM_INC -
PKG_CONFIG_PATH -
LANGUAGE -
LC_ADDRESS -
LC_NAME -
LC_MONETARY -
CONNECT_FB_PASSWORD -
PYENV_VERSION -
KRB5CCNAME -
PWD -
PYENV_VIRTUALENV_INIT -
LOGNAME -
XDG_SESSION_TYPE -
MOTD_SHOWN -
HOME -
LANG -
LC_PAPER -
LS_COLORS -
SSH_CONNECTION -
PYENV_DIR -
CONNECT_UI_PASSWORD -
NVM_DIR -
CONNECT_YT_USER -
LESSCLOSE -
XDG_SESSION_CLASS -
TERM -
LC_IDENTIFICATION -
LESSOPEN -
LIBVIRT_DEFAULT_URI -
USER -
CONNECT_UI_URL -
SHLVL -
NVM_CD_FLAGS -
LC_TELEPHONE -
LC_MEASUREMENT -
XDG_SESSION_ID -
PAPERSIZE -
XDG_RUNTIME_DIR -
SSH_CLIENT -
PYENV_ROOT -
LC_TIME -
XDG_DATA_DIRS -
PATH -
DBUS_SESSION_BUS_ADDRESS -
NVM_BIN -
SSH_TTY -
LC_NUMERIC -
OLDPWD -
PIP_SHIMS_BASE_MODULE -
PIP_DISABLE_PIP_VERSION_CHECK -
PYTHONDONTWRITEBYTECODE -
PIP_PYTHON_PATH -
PYTHONFINDER_IGNORE_UNSUPPORTED
Pipenv–specific environment variables:
Debug–specific environment variables:
-
PATH:/home/SYS/comp1/.pyenv/versions/3.7.13/bin:/home/SYS/comp1/.pyenv/libexec:/home/SYS/comp1/.pyenv/plugins/python-build/bin:/home/SYS/comp1/.pyenv/plugins/pyenv-virtualenv/bin:/home/SYS/comp1/.pyenv/shims:/home/SYS/comp1/.pyenv/versions:/home/SYS/comp1/.pyenv/bin:/home/SYS/comp1/.pyenv/plugins/pyenv-virtualenv/shims:/home/SYS/comp1/.pyenv/bin:/home/SYS/comp1/.local/bin:/home/SYS/comp1/.pyenv/versions:/home/SYS/comp1/.pyenv/bin:/shims:/home/SYS/comp1/.nvm/versions/node/v16.13.0/bin:/home/SYS/comp1/.pyenv/plugins/pyenv-virtualenv/shims:/home/SYS/comp1/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin -
SHELL:/bin/bash -
LANG:en_US.UTF-8 -
PWD:/home/SYS/comp1/git/project/pytest
Contents of Pipfile ('/home/SYS/comp1/git/project/pytest/Pipfile'):
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
pylint = "*"
autopep8 = "*"
[packages]
pytest = "==7.*"
pytest-env = "*"
requests = "==2.*"
configparser = "*"
pytest-repeat = "*"
chromedriver-binary = "*"
pytest-instafail = "*"
mdv = "*"
pyinstaller = "*"
pytest-html-reporter= "*"
matplotlib = "*"
openpyxl = "*"
pytest-html = "==2.1.1"
custom-lib-helpers = {git = '[email protected]/org/custom-lib-helpers.git'}
custom-lib-rest-api = {git = '[email protected]/org/custom-lib-rest-api.git'}
[requires]
python_version = "3.10"
Please check with 2022.7.4 version, the versions you have mentioned are fairly old at this point.
@matteius yes, I have tried the latest version, it's the same issue.
I was testing every version from latest until I discovered that 2020.11.15 works.
@alexanderdevm Can you try these things:
1.) Change the single quotes to double quote:
custom_lib = {git = "[email protected]/org/custom_lib.git"}
2.) Trying adding editable = true as in tis example: requests = {editable = true, git = "https://github.com/psf/requests.git"}
3.) If neither of those things, I am not sure if the ssh git@ version is what is causing you the issue, are you able to specify it as https:// similar to this which did work fine: requests = {editable = true, git = "https://github.com/psf/requests.git"}
@matteius
I have updated back to latest pipenv:
pipenv --version
pipenv, version 2022.7.4
- change to double quotes -> same issue
- added editable -> same issue
- I am not able to use https since the repo needs ssh key to be cloned, fails otherwise.
@alexanderdevm Thanks for checking that, it will have to be investigated further where the code issue may be.
@alexanderdevm So I did a trial and was not able to reproduce it, but I didn't use your setup.py. It seems odd to me in your setup.py that the install_requires has an f-string that appears to require its own self as a dependency. I wonder if that is what is causing confusion in requirementslib? I am not entirely sure what the specific issue is, but it clearly seems to be the way something is defined in the setup.py.
Here is my example of the ssh:// working:
(requirementslib) matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5167$ pipenv lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (7fff55)!
(requirementslib) matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5167$ pipenv sync
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing dependencies from Pipfile.lock (7fff55)...
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:06
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
All dependencies are now up-to-date!
(requirementslib) matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5167$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
redis = {editable = true, git = "[email protected]:matteius/redis-py.git"}
[dev-packages]
[requires]
python_version = "3.9"
(requirementslib) matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5167$ cat Pipfile.lock
{
"_meta": {
"hash": {
"sha256": "f68e9c086eb8b6f263d0820ecb7b6d7a0e45bd7912ec2337a907d9d9ca7fff55"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.9"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.org/simple",
"verify_ssl": true
}
]
},
"default": {
"redis": {
"editable": true,
"git": "ssh://[email protected]/matteius/redis-py.git",
"ref": "d811ae71dbdbeeb4fc0ee73a96b7fbdb1aec8522"
}
},
"develop": {}
}
Yeah it has to be the custom_lib@git+ssh://[email protected]/org/custom_lib.git#egg=custom_lib that is causing issues, from the setup.py
Well to my surprise, I adapted my redis-py example for your setup.py and it does install ok in that format.
from setuptools import setup
HOST = 'git+ssh://[email protected]/matteius/'
setup(
author='Team',
name='custom_lib',
version='2.0.4',
description='Custom Lib',
install_requires=[f'redis@{HOST}redis-py.git#egg=redis'],
py_modules=['custom_lib']
)
(requirementslib) matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5167$ pip install -e .
Obtaining file:///home/matteius/pipenv-triage/pipenv-5167
Preparing metadata (setup.py) ... done
Collecting redis@ git+ssh://[email protected]/matteius/redis-py.git#egg=redis
Cloning ssh://****@github.com/matteius/redis-py.git to /tmp/pip-install-0xgbf4vb/redis_3381f1a1184e40f9bc40f01d0e5eb136
Running command git clone --filter=blob:none --quiet 'ssh://****@github.com/matteius/redis-py.git' /tmp/pip-install-0xgbf4vb/redis_3381f1a1184e40f9bc40f01d0e5eb136
Resolved ssh://****@github.com/matteius/redis-py.git to commit d811ae71dbdbeeb4fc0ee73a96b7fbdb1aec8522
Preparing metadata (setup.py) ... done
Building wheels for collected packages: redis
Building wheel for redis (setup.py) ... done
Created wheel for redis: filename=redis-3.3.8-py2.py3-none-any.whl size=66282 sha256=05614a036e8143e3bf18a5d7dbd788c34a7e8b59cd7346861b2652ff99454124
Stored in directory: /tmp/pip-ephem-wheel-cache-g66i9r4f/wheels/8e/99/45/0ae8832dbcfc83c5032262c66f677a4b9d30cd8214f11ccec5
Successfully built redis
Installing collected packages: redis, custom-lib
Attempting uninstall: redis
Found existing installation: redis 4.3.4
Uninstalling redis-4.3.4:
Successfully uninstalled redis-4.3.4
Attempting uninstall: custom-lib
Found existing installation: custom-lib 2.0.4
Uninstalling custom-lib-2.0.4:
Successfully uninstalled custom-lib-2.0.4
Running setup.py develop for custom-lib
Successfully installed custom-lib-2.0.4 redis-3.3.8
(requirementslib) matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5167$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
redis = {editable = true, git = "[email protected]:matteius/redis-py.git"}
[dev-packages]
[requires]
python_version = "3.9"
(requirementslib) matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5167$ cat Pipfile.lock
{
"_meta": {
"hash": {
"sha256": "f68e9c086eb8b6f263d0820ecb7b6d7a0e45bd7912ec2337a907d9d9ca7fff55"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.9"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.org/simple",
"verify_ssl": true
}
]
},
"default": {
"redis": {
"editable": true,
"git": "ssh://[email protected]/matteius/redis-py.git",
"ref": "d811ae71dbdbeeb4fc0ee73a96b7fbdb1aec8522"
}
},
"develop": {}
}
@alexanderdevm What version of pip are you on? pip --version
I suspect now based on your original output that you are not on a version of pip compatible with the newer pipenv versions. Since 2022.7.4 has vendor'd pip==22.0.4 I would suggest try installing that or newer to see if you get different results.
@matteius
pip --version
pip 22.1.2
i will do additional testing and see if I can create a public way to repro the issue.
@matteius
I have created repo to be able reproduce the issue github: https://github.com/alexanderdevm/pipenv-bug-repro
Steps:
git clone https://github.com/alexanderdevm/pipenv-bug-repro.git
cd pipenv-bug-repro
pipenv install
@alexanderdevm The problem seems to be that the setup.py is not installable the way its written for https://github.com/alexanderdevm/test-lib-1
git clone https://github.com/alexanderdevm/test-lib-1.git
cd test-lib-1
python setup.py install
Installed /home/matteius/.virtualenvs/requirementslib/lib/python3.9/site-packages/test_lib_1-1.0.0-py3.9.egg
Processing dependencies for test-lib-1==1.0.0
Searching for test-lib-2@ git+https://github.com/alexanderdevm/test-lib-2.git#egg=test-lib-2
Reading https://pypi.org/simple/test-lib-2/
Couldn't find index page for 'test-lib-2' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
No local packages or working download links found for test-lib-2@ git+https://github.com/alexanderdevm/test-lib-2.git#egg=test-lib-2
error: Could not find suitable distribution for Requirement.parse('test-lib-2@ git+https://github.com/alexanderdevm/test-lib-2.git#egg=test-lib-2')
@matteius
I have did additional testing and the issue is comes from the following:
install_requires=[f'test-lib-2@{HOST}test-lib-2.git#egg=test-lib-2'],
if I use a variable in the name of install_requires, then it fails with AttributeError: 'JoinedStr', if I change it from f-string to string concat, it will complain about that too.
the only time it installs without an issue is if I dont use any variables, and this issue has started after release 2020.11.15
@alexanderdevm Can you share an example setup.py that works without issues? I tried this but I still get that setuptools is trying to find it in pypi.
'''
Test lib 1
'''
from setuptools import setup
setup(
author='Demo',
name='test-lib-1',
version='1.0.0',
description='Test lib 1',
install_requires=['test-lib-2@git+https://github.com/alexanderdevm/test-lib-2#egg=test-lib-2'],
py_modules=['test-lib-1']
)
(requirementslib) matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5167/test-lib-1$ cat setup.py
Installed /home/matteius/.virtualenvs/requirementslib/lib/python3.9/site-packages/test_lib_1-1.0.0-py3.9.egg
Processing dependencies for test-lib-1==1.0.0
Searching for test-lib-2@ git+https://github.com/alexanderdevm/test-lib-2#egg=test-lib-2
Reading https://pypi.org/simple/test-lib-2/
Couldn't find index page for 'test-lib-2' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
No local packages or working download links found for test-lib-2@ git+https://github.com/alexanderdevm/test-lib-2#egg=test-lib-2
error: Could not find suitable distribution for Requirement.parse('test-lib-2@ git+https://github.com/alexanderdevm/test-lib-2#egg=test-lib-2')
I believe this is an issue with setuptools but you are suggesting you found a way to get it to work without string interpolation and I haven't found a way to get the example to work with a static string. You may need to make a report to setuptools about it.
Discussions that seem relevant: https://github.com/pypa/setuptools/issues/1740#issuecomment-482377926 https://github.com/pypa/setuptools/issues/1052
I did from reading that first discussion gather that python setup.py install will never work now with a install_requires that has a git+ repostiory, but that pip itself can handle it. I verified it that pip can do an editable install of the above setup.py that setuptools outright fails to install.
(requirementslib) matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5167$ pip install -e test-lib-1/
Obtaining file:///home/matteius/pipenv-triage/pipenv-5167/test-lib-1
Preparing metadata (setup.py) ... done
Collecting test-lib-2@ git+https://github.com/alexanderdevm/test-lib-2#egg=test-lib-2
Cloning https://github.com/alexanderdevm/test-lib-2 to /tmp/pip-install-rgtj5xrw/test-lib-2_954b2f28777748cb94c38921bda977aa
Running command git clone --filter=blob:none --quiet https://github.com/alexanderdevm/test-lib-2 /tmp/pip-install-rgtj5xrw/test-lib-2_954b2f28777748cb94c38921bda977aa
Resolved https://github.com/alexanderdevm/test-lib-2 to commit 877debd2847747ea440468fcd5cc00f300d124ab
Preparing metadata (setup.py) ... done
Building wheels for collected packages: test-lib-2
Building wheel for test-lib-2 (setup.py) ... done
Created wheel for test-lib-2: filename=test_lib_2-1.0.0-py3-none-any.whl size=1144 sha256=a0e19718d4db52a5fbe42b188a371f8248ffa60aa62725727872cbf21f7f4dff
Stored in directory: /tmp/pip-ephem-wheel-cache-izrouwo_/wheels/0a/df/b5/b049f0899c6f0464bfaed9960dd17e0656b84fa093a58124b3
Successfully built test-lib-2
Installing collected packages: test-lib-2, test-lib-1
Attempting uninstall: test-lib-1
Found existing installation: test-lib-1 1.0.0
Uninstalling test-lib-1-1.0.0:
Successfully uninstalled test-lib-1-1.0.0
Running setup.py develop for test-lib-1
Successfully installed test-lib-1-1.0.0 test-lib-2-1.0.0
When I changed setup.py back to string interpolation and ran pip install -e test-lib-1/ It also worked.
@alexanderdevm Actually from my last message I found that setup.py install doesn't support VCS install_requires but that pip does. It seems your bug is in requirements lib. I patched the vendor version of pipenv requirements lib locally and got it to seemingly work, but I am not convinced of my logic being correct.
It seems that an f-string is ast parsed as an ast.JoinedStr The problem with my patch is I am not actually sure what can convert an ast.JoinedStr to an actual string. I print the result of what I am appending to the list and it prints:
<ast.JoinedStr object at 0x00000218B1BF68F0>
<ast.JoinedStr object at 0x00000218B1BF4E20>
Yet I append the ast.JoinedStr to the result list and it gets past your error and seems to work with this patch:
$ git diff
diff --git a/pipenv/vendor/requirementslib/models/setup_info.py b/pipenv/vendor/requirementslib/models/setup_info.py
index 4acd6d6a..2af24437 100644
--- a/pipenv/vendor/requirementslib/models/setup_info.py
+++ b/pipenv/vendor/requirementslib/models/setup_info.py
@@ -356,7 +356,14 @@ class SetupReader:
return []
if isinstance(value, ast.List):
- return [el.s for el in value.elts]
+ result = []
+ for el in value.elts:
+ if isinstance(el, ast.JoinedStr):
+ print(el)
+ result.append(el)
+ else:
+ result.append(el.s)
+ return result
elif isinstance(value, ast.Name):
variable = cls._find_variable_in_body(body, value.id)
I am not convinced this is the right way to handled the ast.JoinedStr but I am convinced this is a bug in requirementslib we can patch somehow -- if you would mind opening an issue report there @alexanderdevm would be helpful.
After exploring this further, I am not convinced there is a safe way to evaluate the f-string or format string from the setup.py. The way that requirementslib is implementing the class SetupReader: is a """Class that reads a setup.py file without executing it.""".
The problem is one of safety -- if we evaluated the logic of setup.py in the resolution of dependencies there could be a malicious package in the chain that is going to execute unexpected code behind the scenes. So it appears the way it is written is not very possible to return the evaluated string from requirementslib ... Where it gets complicated is I tried just returning the AST for the JoinedStr and somehow in my test from pipenv it worked out and generated an appropriate Pipfile and Pipfile.lock but from the perspective of requirementslib I am not passing my own test expectations -- it is not clear where pipenv would be evaluating the AST, maybe in pyparsing, but somehow with my change to requirementslib it gets past the initial failures and then I am guessing the setup.py and/or the AST gets evaluated anyway at a later part of the process. My failing test I added shows that it is just returning an AST expression and not the actual string -- because the SetupReader is intentionally reading the AST of the file and the only strings it can return are the things that are already final string value which don't require computation. Even "String %s" % "Other String" won't be evaluated in this case.
@matteius thank you for looking into the issue. I have created a bug with requirementslib https://github.com/sarugaku/requirementslib/issues/328
The next step is a fix requirementslib side?
@alexanderdevm I think we can make that case a better error, but I don't think we can safely evaluate an f-string or any interpolated string from the setup.py. I think you will need to consider alternative ways to handling that setup.py install_requires differently. I am not 100% sure how my patch gets evaluated in pipenv, it seemed to work, but there was a weird side effect where it looked for: https://files.pythonhosted.org/packages/b6/38/1b2188bea6b5346ea2f97f063c99fdadb36707a7b3a95ff4fe73e242c33c/ast which I think was the <_ast.JoinedStr object at 0x7f2aa03eb190> type object that was returned from my patch of requirementslib.
Plus the setup.py the way it is written also is not compatible with setuptools for you cannot run python setup.py install and have it work, and I think that is the primary function for having the setup.py. It seems that pip's ability to do it is quite the special case, but at that point you are telling pip to definitely install whatever, but I think we wouldn't want to invoke and evaluate the setup.py in a resolution phase due to dangers that you call out a git repository somewhere and the setup.py changes to contain malicious code that gets evaluated in a lock cycle.
@matteius , a suggestion is to remove variables from install_requires for the time being?
Its also interesting that version of pipenv 2020 and below had no issue.
Actually my all time favourite version of pipenv is 2018.11.26 this is the version that generates lock files the fastest when using git packages.
a suggestion is to remove variables from install_requires for the time being?
Correct
I believe that the 2018 version is still on pip-tools and is going to be lacking quite a few security minded and other types of fixes. I have only been using pipenv for 9 months now, so I cannot speak to the older versions but I know we have been making a lot of enhancements and bug fixes in recent versions. I suspect from looking at the code that it was determined to be a security vulnerability the way it was written prior and it was in 2021 that the class SetupReader: was added to requirementslib which is self described as a """Class that reads a setup.py file without executing it.""".
Its not entirely clear to me what happened from 2020.11.15 -> 2021.5.29 that would have went from working to not working for your example: https://github.com/pypa/pipenv/compare/v2020.11.15...v2021.5.29#diff-9d03f8495a67df8e88527db31ae4040a36cfa0fbb73b2fd0d6ef0174d7066045
@matteius I have updated all my setup.py to be without f-string for the moment and that resolves the issue .
For the moment the next course of action:
- Pipenv : improve error message for f-string, string concat,... errors
- requirementslib: wait for update on possible patch/fix for this issue?
Thanks @alexanderdevm -- I think that the best we will be able to do is throw a better user friendly error in requirementslib that explains the issue. Please see this article for more context on why we don't want to execute arbitrary code, even though pip will do it if you go to install, that seems like a side-effect of the design and not desired. This isn't the exact article I was trying to find you, but it talks about this issue too: https://jfrog.com/blog/python-wheel-jacking-in-supply-chain-attacks/
I just ran into the same kind of issue as described here with:
AttributeError: 'JoinedStr' object has no attribute 's'
As a workaround, I found that you can wrap your install_requires in a list, which does not really do anything since it is already a list, but it fixes the issue. I assume it just evaluates the list items before the error occurs.
setup(
...
install_requires=list(['easysnmp', f'custom_lib@{HOST}custom_lib.git#egg=custom_lib']),
...
)
I initially had an issue with using if-else within install_requires, which resulted in the following error:
AttributeError: 'IfExp' object has no attribute 's'
The workaround also fixes this issue. Example:
TF_VERSION = '2.3.0'
setup(
...
install_requires = list([
f'tensorflow-macos>={TF_VERSION}' if sys.platform == 'darwin' and platform.machine() == 'arm64' else f'tensorflow>={TF_VERSION}']),
...
)
We dropped requirementslib and I believe this error should be improved now.