pipenv icon indicating copy to clipboard operation
pipenv copied to clipboard

Published 20 hours ago • verified publisher icon pypa

Installing a package from VCS fails with `Invalid specifier: '==attr: ...` if the package is already installed

Open Jmennius opened this issue 2 years ago • 2 comments

Issue description

When trying install/lock a project from VCS that uses version = attr: <package.field> specifier in setup.cfg, if the package was (is) previously installed from a wheel, I get an error: pipenv.patched.pip._vendor.packaging.specifiers.InvalidSpecifier: Invalid specifier: '==attr: PyInstaller.__version__'

if installing from VCS when the package is not installed from the wheel yet - the issue does not happen.

Expected result

No error occurs, package is installed and locked.

Actual result

$ pipenv --python ~/.pyenv/versions/3.8.13/bin/python3 --verbose install git+https://github.com/pyinstaller/pyinstaller.git@develop#egg=pyinstaller 
Installing git+https://github.com/pyinstaller/pyinstaller.git@develop#egg=pyinstaller...
Installing package: git+https://github.com/pyinstaller/pyinstaller.git@develop#egg=pyinstaller
Writing supplied requirement line to temporary file: 'git+https://github.com/pyinstaller/pyinstaller.git@develop#egg=pyinstaller'
Installing 'pyinstaller'
$ /var/home/ievgenp/src/pipenv-test/.venv/bin/python /var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/patched/pip/__pip-runner__.py install --no-input --verbose --upgrade --exists-action=i -r /tmp/pipenv-80p7itc2-requirements/pipenv-wvj33x41-requirement.txt -i https://pypi.org/simple
Using source directory: '/var/home/ievgenp/src/pipenv-test/.venv/src'
Adding pyinstaller to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock (6f7040) out of date, updating to (2c5d04)...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Reporter.starting()
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('altgraph'), None)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('altgraph'), None)
Reporter.adding_requirement(SpecifierRequirement('pyinstaller-hooks-contrib>=2021.4'), None)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('pyinstaller-hooks-contrib>=2021.4'), None)
Reporter.adding_requirement(SpecifierRequirement('setuptools'), None)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('setuptools'), None)
Reporter.starting_round(0)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(0)
Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/84/3f/1a5c9bef54cac9bf41edd6f4aaf61cd52ed578e10ccc607e0278012cb4a5/altgraph-0.17.2-py2.py3-none-any.whl (from https://pypi.org/simple/altgraph/)'))
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/84/3f/1a5c9bef54cac9bf41edd6f4aaf61cd52ed578e10ccc607e0278012cb4a5/altgraph-0.17.2-py2.py3-none-any.whl (from https://pypi.org/simple/altgraph/)'))
Reporter.ending_round(0, state)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending_round(0, state)
Reporter.starting_round(1)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(1)
Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/33/39/803aed3f31826e308cd6391902b78d1f3375a4191d1a5d9c84a315e5f55e/pyinstaller_hooks_contrib-2022.10-py2.py3-none-any.whl (from https://pypi.org/simple/pyinstaller-hooks-contrib/) (requires-python:>=3.7)'))
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/33/39/803aed3f31826e308cd6391902b78d1f3375a4191d1a5d9c84a315e5f55e/pyinstaller_hooks_contrib-2022.10-py2.py3-none-any.whl (from https://pypi.org/simple/pyinstaller-hooks-contrib/) (requires-python:>=3.7)'))
Reporter.ending_round(1, state)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending_round(1, state)
Reporter.starting_round(2)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(2)
Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/d9/5f/2daccd14278b6b780ae6799f85998377c06019354982391245f4b58a927d/setuptools-65.3.0-py3-none-any.whl (from https://pypi.org/simple/setuptools/) (requires-python:>=3.7)'))
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/d9/5f/2daccd14278b6b780ae6799f85998377c06019354982391245f4b58a927d/setuptools-65.3.0-py3-none-any.whl (from https://pypi.org/simple/setuptools/) (requires-python:>=3.7)'))
Reporter.ending_round(2, state)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending_round(2, state)
Reporter.starting_round(3)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(3)
Reporter.ending(State(mapping=OrderedDict([('altgraph', LinkCandidate('https://files.pythonhosted.org/packages/84/3f/1a5c9bef54cac9bf41edd6f4aaf61cd52ed578e10ccc607e0278012cb4a5/altgraph-0.17.2-py2.py3-none-any.whl (from https://pypi.org/simple/altgraph/)')), ('pyinstaller-hooks-contrib', LinkCandidate('https://files.pythonhosted.org/packages/33/39/803aed3f31826e308cd6391902b78d1f3375a4191d1a5d9c84a315e5f55e/pyinstaller_hooks_contrib-2022.10-py2.py3-none-any.whl (from https://pypi.org/simple/pyinstaller-hooks-contrib/) (requires-python:>=3.7)')), ('setuptools', LinkCandidate('https://files.pythonhosted.org/packages/d9/5f/2daccd14278b6b780ae6799f85998377c06019354982391245f4b58a927d/setuptools-65.3.0-py3-none-any.whl (from https://pypi.org/simple/setuptools/) (requires-python:>=3.7)'))]), criteria={'altgraph': Criterion((SpecifierRequirement('altgraph'), via=None)), 'pyinstaller-hooks-contrib': Criterion((SpecifierRequirement('pyinstaller-hooks-contrib>=2021.4'), via=None)), 'setuptools': Criterion((SpecifierRequirement('setuptools'), via=None))}, backtrack_causes=[]))
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending(State(mapping=OrderedDict([('altgraph', LinkCandidate('https://files.pythonhosted.org/packages/84/3f/1a5c9bef54cac9bf41edd6f4aaf61cd52ed578e10ccc607e0278012cb4a5/altgraph-0.17.2-py2.py3-none-any.whl (from https://pypi.org/simple/altgraph/)')), ('pyinstaller-hooks-contrib', LinkCandidate('https://files.pythonhosted.org/packages/33/39/803aed3f31826e308cd6391902b78d1f3375a4191d1a5d9c84a315e5f55e/pyinstaller_hooks_contrib-2022.10-py2.py3-none-any.whl (from https://pypi.org/simple/pyinstaller-hooks-contrib/) (requires-python:>=3.7)')), ('setuptools', LinkCandidate('https://files.pythonhosted.org/packages/d9/5f/2daccd14278b6b780ae6799f85998377c06019354982391245f4b58a927d/setuptools-65.3.0-py3-none-any.whl (from https://pypi.org/simple/setuptools/) (requires-python:>=3.7)'))]), criteria={'altgraph': Criterion((SpecifierRequirement('altgraph'), via=None)), 'pyinstaller-hooks-contrib': Criterion((SpecifierRequirement('pyinstaller-hooks-contrib>=2021.4'), via=None)), 'setuptools': Criterion((SpecifierRequirement('setuptools'), via=None))}, backtrack_causes=[]))
Traceback (most recent call last):
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/patched/pip/_vendor/packaging/specifiers.py", line 634, in __init__
    parsed.add(Specifier(specifier))
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/patched/pip/_vendor/packaging/specifiers.py", line 98, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
pipenv.patched.pip._vendor.packaging.specifiers.InvalidSpecifier: Invalid specifier: '==attr: PyInstaller.__version__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/resolver.py", line 867, in <module>
    main()
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/resolver.py", line 853, in main
    _main(
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/resolver.py", line 833, in _main
    resolve_packages(
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/resolver.py", line 794, in resolve_packages
    results = clean_results(results, resolver, project, dev)
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/resolver.py", line 659, in clean_results
    entry_dict = translate_markers(entry.get_cleaned_dict(keep_outdated=False))
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/resolver.py", line 279, in get_cleaned_dict
    self.validate_constraints()
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/resolver.py", line 575, in validate_constraints
    pinned_version = self.updated_version
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/resolver.py", line 641, in __getattribute__
    return super().__getattribute__(key)
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/resolver.py", line 423, in updated_version
    version = self.entry.specifiers
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2591, in specifiers
    and self.line_instance.setup_info
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 808, in setup_info
    self.setup_info = self.get_setup_info()
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 821, in setup_info
    self.specifier = setup_info.version
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 414, in specifier
    self.specifiers = SpecifierSet(spec)
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/patched/pip/_vendor/packaging/specifiers.py", line 636, in __init__
    parsed.add(LegacySpecifier(specifier))
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/patched/pip/_vendor/packaging/specifiers.py", line 253, in __init__
    super().__init__(spec, prereleases)
  File "/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv/patched/pip/_vendor/packaging/specifiers.py", line 98, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
pipenv.patched.pip._vendor.packaging.specifiers.InvalidSpecifier: Invalid specifier: '==attr: PyInstaller.__version__'
✘ Locking Failed!

Steps to replicate

pipenv install pyinstaller
pipenv --verbose install git+https://github.com/pyinstaller/pyinstaller.git@develop#egg=pyinstaller

or

pipenv lock  # create an empty venv with pipenv
pipenv run pip install pyinstaller  # install pyinstaller from a wheel into a venv
pipenv install git+https://github.com/pyinstaller/pyinstaller.git@develop#egg=pyinstaller  # try installing from a VCS
$ pipenv --support

Pipenv version: '2022.9.8'

Pipenv location: '/var/home/ievgenp/.pyenv/versions/3.8.13/lib/python3.8/site-packages/pipenv'

Python location: '/var/home/ievgenp/.pyenv/versions/3.8.13/bin/python3.8'

OS Name: 'posix'

User pip version: '22.2.2'

user Python installations found:

  • 3.10.4: /var/home/ievgenp/.pyenv/versions/3.10.4/bin/python3
  • 3.8.13: /var/home/ievgenp/.pyenv/versions/3.8.13/bin/python3.8
  • 3.8.13: /var/home/ievgenp/.pyenv/versions/3.8.13/bin/python3
  • 3.8.13: /var/home/ievgenp/.pyenv/versions/3.8.13/bin/python
  • 3.8.13: /var/home/ievgenp/.pyenv/versions/3.8.13/bin/python3.8
  • 3.8.6: /var/home/ievgenp/.pyenv/versions/3.8.6/bin/python3.8
  • 3.8.5: /var/home/ievgenp/.pyenv/versions/3.8.5/bin/python3.8
  • 3.7.13: /var/home/ievgenp/.pyenv/versions/3.7.13/bin/python3.7m
  • 3.7.9: /var/home/ievgenp/.pyenv/versions/3.7.9/bin/python3.7m
  • 3.7.1: /var/home/ievgenp/.pyenv/versions/3.7.1/bin/python
  • 3.7.1: /var/home/ievgenp/.pyenv/versions/test1/bin/python
  • 3.6.9: /usr/bin/python3.6
  • 3.6.9: /usr/bin/python3.6m
  • 3.6.9: /usr/bin/python3
  • 2.7.17: /usr/bin/python2
  • 2.7.17: /usr/bin/python2.7
  • 2.7.17: /usr/bin/python

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.13',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.19.8-200.fc36.x86_64',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT_DYNAMIC Thu Sep 8 19:02:21 UTC 2022',
 'python_full_version': '3.8.13',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

  • PIPENV_VENV_IN_PROJECT
  • LD_LIBRARY_PATH
  • XDG_MENU_PREFIX
  • P9K_SSH
  • LANG
  • DISPLAY
  • HOSTNAME
  • PYENV_ROOT
  • OLDPWD
  • KEYTIMEOUT
  • PIPENV_SHELL_FANCY
  • DEBFULLNAME
  • COLORTERM
  • _P9K_TTY
  • PYTHON_CONFIGURE_OPTS
  • PYENV_HOOK_PATH
  • SSH_AUTH_SOCK
  • MINICOM
  • container
  • ZDOTDIR
  • USER
  • PYENV_DIR
  • DESKTOP_SESSION
  • VERSION
  • WAYLAND_DISPLAY
  • PWD
  • HOME
  • PYENV_VERSION
  • NAME
  • XDG_SESSION_TYPE
  • BUILDAH_LAYERS
  • XDG_DATA_DIRS
  • XDG_SESSION_DESKTOP
  • CMAKE_BUILD_PARALLEL_LEVEL
  • TERM
  • SHELL
  • VTE_VERSION
  • DOCKER_HOST
  • USER_CERTIFICATE
  • USER_PRIVATE_KEY
  • XDG_CURRENT_DESKTOP
  • PYENV_SHELL
  • SHLVL
  • TOOLBOX_PATH
  • DEBEMAIL
  • LOGNAME
  • DBUS_SESSION_BUS_ADDRESS
  • XDG_RUNTIME_DIR
  • XAUTHORITY
  • PATH
  • P9K_TTY
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PIP_PYTHON_PATH
  • PYTHONDONTWRITEBYTECODE
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

  • PIPENV_VENV_IN_PROJECT: 1
  • PIPENV_SHELL_FANCY: 1

Debug–specific environment variables:

  • PATH: /var/home/ievgenp/.pyenv/versions/3.8.13/bin:/var/home/ievgenp/.pyenv/libexec:/var/home/ievgenp/.pyenv/plugins/python-build/bin:/var/home/ievgenp/go/bin:/var/home/ievgenp/.pyenv/shims:/var/home/ievgenp/.pyenv/bin:/var/home/ievgenp/.local/bin:/var/home/ievgenp/.cargo/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/home/ievgenp/.local/share/flatpak/exports/bin:/var/lib/flatpak/exports/bin
  • SHELL: /usr/bin/zsh
  • LANG: en_US.UTF-8
  • PWD: /var/home/ievgenp/src/pipenv-test

Contents of Pipfile ('/var/home/ievgenp/src/pipenv-test/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pyinstaller = {ref = "develop", git = "https://github.com/pyinstaller/pyinstaller.git"}

[dev-packages]

[requires]
python_version = "3.8"

Contents of Pipfile.lock ('/var/home/ievgenp/src/pipenv-test/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "7f7606f08e0544d8d012ef4d097dabdd6df6843a28793eb6551245d4b2db4242"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

Jmennius avatar Sep 16 '22 10:09 Jmennius

@Jmennius I was able to reproduce your issue -- I think it has to do with I don't see a version specifier in the setup.py for pyinstaller -- https://github.com/pyinstaller/pyinstaller/blob/develop/setup.py#L253-L272

This confuses the resolver because it needs to pin to a specific version I believe.

matteius avatar Sep 20 '22 18:09 matteius

@Jmennius I was able to reproduce your issue -- I think it has to do with I don't see a version specifier in the setup.py for pyinstaller -- https://github.com/pyinstaller/pyinstaller/blob/develop/setup.py#L253-L272

This confuses the resolver because it needs to pin to a specific version I believe.

But the version specifier is in setup.cfg - shouldn't it be enough? Plus, it works in some cases - like if you install the package in a clean environment.

Jmennius avatar Sep 20 '22 19:09 Jmennius

It looks like behaviour changed in setuptools: https://github.com/pypa/setuptools/issues/1724

And per the documentation this should be working now: https://packaging.python.org/en/latest/guides/single-sourcing-package-version/

So this means pipenv is using an old setuptools version?

Anyway, I had to hardcode versions in every setup.cfg of my packages to prevent this:

[metadata]
name = package
version = v0.0.1

This is undesirable package mantaining workflow but it's what's allowing me to use pipenv install -e . without failing

iuriguilherme avatar Oct 12 '22 03:10 iuriguilherme

@iuriguilherme Pipenv only calls out "setuptools>=36.2.1" in order to support python 3.7 I believe, or maybe this was for 3.6 and the pin could be increased. Either way, the end user can install whichever version of setuptools they want that is at least that new.

matteius avatar Dec 09 '22 09:12 matteius

The issue I was having with PyInstaller was fixed via pyinstaller/pyinstaller#7444 for me. I am not really sure if it's a bug in setuptools or if it's just improper usage of declarative/imperative setup... Unless anybody comments in a few weeks I will close this from my side as fixed.

Jmennius avatar Feb 20 '23 08:02 Jmennius

@Jmennius We bumped the setuptools version requirement with the new pipenv release, so unless a new issue crops up, I think we can close this for now. Can defer to you for the next week or two.

matteius avatar Feb 20 '23 09:02 matteius 

src/
  package/
      __init__.py
      __main__.py
      _version.py
setup.cfg

__init__.py:

from ._version import __version__
__name__: str = "package"
__description__: str = "No one reads this"

_version.py:

__version__: str = 0.0.0.1

setup.cfg:

[metadata]
name = attr: package.__name__
version = attr: package._version.__version__
description = attr: package.__description__

Working for me.

iuriguilherme avatar Feb 23 '23 07:02 iuriguilherme