pipenv icon indicating copy to clipboard operation
pipenv copied to clipboard

Published 20 hours ago • verified publisher icon pypa

pipenv fails to parse platform-specific requirement in Pipfile when using a versioning scheme

Open OliverFarren opened this issue 1 year ago • 4 comments

Issue description

When installing a package with a platform specifier for windows whilst on mac, along with a package version like the following

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
python-magic = "==0.4.*"
python-magic-bin = { version = "==0.4.*", markers="sys_platform == 'win32'" }

[dev-packages]

[requires]
python_version = "3.11"
python_full_version = "3.11.1"

i'm getting the following error

pipenv.patched.pip._internal.exceptions.InstallationError: Invalid requirement: 'python-magic-bin0.4.*'

It feels like there's a parsing issue. If I wildcard the version it works fine and it appears the == has been stripped away. What's also weird to me though is that in the stdout output it clearly skips the dependency due to it being 'win32'.

Non verbose output:

(pipfile-bug) ➜  pipfile-bug pipenv install
Pipfile.lock (9df141) out of date: run `pipfile lock` to update to (4a8526)...
Running $ pipenv lock then $ pipenv sync.
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
⠙ Locking packages...Warning: Could not find a matching version of python-magic-bin==0.4.*; sys_platform == "win32" for your environment, its dependencies will be skipped.

Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/pyparsing/util.py", line 256, in _inner
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/pyparsing/core.py", line 1190, in parse_string
    raise exc.with_traceback(None)
pipenv.patched.pip._vendor.pyparsing.exceptions.ParseException: Expected string_end, found '.'  (at char 19), (line:1, col:20)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 371, in _parse_req_string
    req = get_requirement(req_as_string)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_internal/utils/packaging.py", line 45, in get_requirement
    return Requirement(req_string)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 104, in __init__
    raise InvalidRequirement(
pipenv.patched.pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'.*'": Expected string_end

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/install.py", line 93, in do_install
    do_init(
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/install.py", line 624, in do_init
    do_update(
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/update.py", line 61, in do_update
    do_lock(
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/lock.py", line 66, in do_lock
    venv_resolve_deps(
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 899, in venv_resolve_deps
    return prepare_lockfile(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/locking.py", line 148, in prepare_lockfile
    lockfile_entry = get_locked_dep(project, dep, pipfile, current_entry)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/locking.py", line 131, in get_locked_dep
    lockfile_entry = clean_resolved_dep(project, dep, is_top_level, current_entry)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 277, in clean_resolved_dep
    potential_hashes = unearth_hashes_for_dep(project, dep)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 189, in unearth_hashes_for_dep
    install_req, markers, _ = install_req_from_pipfile(dep["name"], dep)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 1097, in install_req_from_pipfile
    install_req, _ = expansive_install_req_from_line(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 996, in expansive_install_req_from_line
    parts = parse_req_from_line(pip_line, line_source)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 399, in parse_req_from_line
    req: Optional[Requirement] = _parse_req_string(req_as_string)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 385, in _parse_req_string
    raise InstallationError(msg)
pipenv.patched.pip._internal.exceptions.InstallationError: Invalid requirement: 'python-magic-bin0.4.*'

Verbose output

pipfile-bug pipenv install --verbose
Pipfile.lock (9df141) out of date: run `pipfile lock` to update to (51a332)...
Running $ pipenv lock then $ pipenv sync.
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Could not find a matching version of python-magic-bin==0.4.*; sys_platform == "win32" for your environment, its dependencies will be skipped.
INFO:pipenv.patched.pip._internal.resolution.resolvelib.factory:Ignoring python-magic-bin: markers 'sys_platform == "win32"' don't match your environment
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting()
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('python-magic==0.4.*'), None)
INFO:pipenv.patched.pip._internal.operations.prepare:Collecting python-magic==0.4.* (from -r 
/var/folders/n4/hqk2v4vx52qbpt0h8cl_3__c0000gn/T/pipenv-8l0qtcv3-requirements/pipenv-2tgqli6m-constraints.txt (line 2))
INFO:pipenv.patched.pip._internal.network.download:Using cached python_magic-0.4.27-py2.py3-none-any.whl.metadata (5.8 kB)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(0)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/6c/73/9f872cb81fc5c3bb48f7227872c28975f998f3e7c2b1c16e
95e6432bbb90/python_magic-0.4.27-py2.py3-none-any.whl (from https://pypi.org/simple/python-magic/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*)'))
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending_round(0, state)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(1)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending(State(mapping=OrderedDict([('python-magic', 
LinkCandidate('https://files.pythonhosted.org/packages/6c/73/9f872cb81fc5c3bb48f7227872c28975f998f3e7c2b1c16e95e6432bbb90/python_magic-0.4.27-py2.py3-none-any.whl (from 
https://pypi.org/simple/python-magic/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*)'))]), criteria={'python-magic': 
Criterion((SpecifierRequirement('python-magic==0.4.*'), via=None))}, backtrack_causes=[]))
INFO:pipenv.patched.pip._internal.network.download:Using cached python_magic-0.4.27-py2.py3-none-any.whl (13 kB)
⠹ Locking packages...
✔ Success!
Warning: Error generating hash for python-magic-bin.
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/pyparsing/util.py", line 256, in _inner
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/pyparsing/core.py", line 1190, in parse_string
    raise exc.with_traceback(None)
pipenv.patched.pip._vendor.pyparsing.exceptions.ParseException: Expected string_end, found '.'  (at char 19), (line:1, col:20)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 367, in _parse_req_string
    req = get_requirement(req_as_string)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_internal/utils/packaging.py", line 45, in get_requirement
    return Requirement(req_string)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 104, in __init__
    raise InvalidRequirement(
pipenv.patched.pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'.*'": Expected string_end

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/install.py", line 93, in do_install
    do_init(
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/install.py", line 624, in do_init
    do_update(
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/update.py", line 61, in do_update
    do_lock(
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/lock.py", line 66, in do_lock
    venv_resolve_deps(
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 899, in venv_resolve_deps
    return prepare_lockfile(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/locking.py", line 148, in prepare_lockfile
    lockfile_entry = get_locked_dep(project, dep, pipfile, current_entry)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/locking.py", line 131, in get_locked_dep
    lockfile_entry = clean_resolved_dep(project, dep, is_top_level, current_entry)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 277, in clean_resolved_dep
    potential_hashes = unearth_hashes_for_dep(project, dep)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 189, in unearth_hashes_for_dep
    install_req, markers, _ = install_req_from_pipfile(dep["name"], dep)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 1097, in install_req_from_pipfile
    install_req, _ = expansive_install_req_from_line(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 996, in expansive_install_req_from_line
    parts = parse_req_from_line(pip_line, line_source)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 395, in parse_req_from_line
    req: Optional[Requirement] = _parse_req_string(req_as_string)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 381, in _parse_req_string
    raise InstallationError(msg)
pipenv.patched.pip._internal.exceptions.InstallationError: Invalid requirement: 'python-magic-bin0.4.*'

Steps to replicate

mkdir pipfile-bug
cd pipfile-bug
pipenv install --python 3.11

Add following to Pipfile

python-magic = "==0.4.*"
python-magic-bin = { version = "==0.4.*", markers="sys_platform == 'win32'" }

Install

pipenv shell
pipenv install
$ pipenv --support

Pipenv version: '2024.0.1'

Pipenv location: '/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2024.0.1/libexec/bin/python'

OS Name: 'posix'

User pip version: '24.0'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.12.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '23.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 23.5.0: Wed May  1 20:09:52 PDT '
                     '2024; root:xnu-10063.121.3~5/RELEASE_X86_64',
 'python_full_version': '3.12.3',
 'python_version': '3.12',
 'sys_platform': 'darwin'}

System environment variables:

  • __CFBundleIdentifier
  • PATH
  • SHELL
  • SECURITYSESSIONID
  • TERM
  • USER
  • COMMAND_MODE
  • TMPDIR
  • LaunchInstanceID
  • TERMINAL_EMULATOR
  • SSH_AUTH_SOCK
  • XPC_FLAGS
  • TERM_SESSION_ID
  • __CF_USER_TEXT_ENCODING
  • LOGNAME
  • LC_CTYPE
  • XPC_SERVICE_NAME
  • HOME
  • SHLVL
  • PWD
  • OLDPWD
  • PYENV_ROOT
  • PYENV_SHELL
  • ZSH
  • PAGER
  • LESS
  • LSCOLORS
  • LS_COLORS
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PIP_PYTHON_PATH
  • PIPENV_ACTIVE
  • VIRTUAL_ENV
  • VIRTUAL_ENV_PROMPT
  • PS1
  • _
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

  • PIPENV_ACTIVE: 1

Debug–specific environment variables:

  • PATH: /Users/oliverfarren/.pyenv/versions/3.9.13/bin:/Users/oliverfarren/.pyenv/versions/3.6.0/bin:/Users/oliverfarren/.pyenv/versions/3.8.13/bin:/Users/oliverfarren/.pyenv/versions/3.11.1/bin:/Users/oliverfarren/.pyenv/versions/3.7.13/bin:/Users/oliverfarren/.pyenv/versions/3.6.15/bin:/Users/oliverfarren/.pyenv/versions/3.10.4/bin:/Users/oliverfarren/.pyenv/versions/3.8.5/bin:/Users/oliverfarren/.pyenv/versions/3.11.4/bin:/Users/oliverfarren/.pyenv/versions/3.8.10/bin:/Users/oliverfarren/.pyenv/versions/3.8.16/bin:/Users/oliverfarren/.pyenv/versions/3.7.16/bin:/Users/oliverfarren/.local/share/virtualenvs/pipfile-bug-h7gOS49K/bin:/Users/oliverfarren/.pyenv/shims:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/TeX/texbin
  • SHELL: /bin/zsh
  • PWD: /Users/oliverfarren/Documents/repos/pipfile-bug
  • VIRTUAL_ENV: /Users/oliverfarren/.local/share/virtualenvs/pipfile-bug-h7gOS49K

Contents of Pipfile ('/Users/oliverfarren/Documents/repos/pipfile-bug/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
python-magic = "==0.4.*"
python-magic-bin = { version = "==0.4.*", markers="sys_platform == 'win32'" }

[dev-packages]

[requires]
python_version = "3.11"
python_full_version = "3.11.1"

Contents of Pipfile.lock ('/Users/oliverfarren/Documents/repos/pipfile-bug/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "da18a2f144044fef4c7f2fe1352f85c1dbfdc6a81298c726b067991b9a9df141"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_full_version": "3.11.1",
            "python_version": "3.11"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

OliverFarren avatar Sep 02 '24 10:09 OliverFarren

@OliverFarren could you re-check with latest pipenv? I was not able to reproduce this one ...

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage
$ cd issue-6231

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-6231
$ pipenv install --python 3.11
Creating a virtualenv for this project
Pipfile: C:\Users\matte\Projects\pipenv-triage\issue-6231\Pipfile
Using C:/Users/matte/.pyenv/pyenv-win/versions/3.11.5/python.exe3.11.5 to create virtualenv...
created virtual environment CPython3.[==  ] 11.5Creating virtual environment....final.0-64 in 3543ms
  creator CPython3Windows(dest=C:\c\users\matte\.virtualenvs\issue-6231-PydLobCd, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\matte\AppData\Local\pypa\virtualenv)
    added seed packages: pip==24.2, setuptools==73.0.1, wheel==0.44.0
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!
Virtualenv location: C:\c\Users\matte\.virtualenvs\issue-6231-PydLobCd
Creating a Pipfile for this project...
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Locking [dev-packages] dependencies...
Updated Pipfile.lock (ace7e9784a57c9841694abb0b34b2c60c71c5820bbe44603aad915ceb02cba3b)!
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Installing dependencies from Pipfile.lock (2cba3b)...

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-6231
$ vim Pipfile

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-6231
$ pipenv shell
Launching subshell in virtual environment...
bash: /c/Users/matte/appdata/Local/Packages/PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0/LocalCache/local-packages/Python310/Scripts/virtualenvwrapper.sh: No such file or directory

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-6231
$ pipenv  install
Pipfile.lock (2cba3b) out of date: run `pipenv lock` to update to (e0502f)...
Running $ pipenv lock then $ pipenv sync.
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (82e322fddcfda772b9a1f273761c0338e140911a70716500e7ae3fc83ae0502f)!
Installing dependencies from Pipfile.lock (e0502f)...
All dependencies are now up-to-date!
Installing dependencies from Pipfile.lock (e0502f)...

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-6231
$ pipenv run pip freeze
python-magic==0.4.27
python-magic-bin==0.4.14

matteius avatar Sep 23 '24 04:09 matteius

Analysis of Pipenv Issue #6231

1. Problem Summary:

The issue describes a problem in Pipenv's parsing of package specifiers within the Pipfile when a versioning scheme (like "==0.4.*") and platform-specific markers are used together. This leads to an InstallationError during the locking process due to the version specifier being incorrectly stripped away and the package name being concatenated with the partial version string.

2. Comment Analysis:

  • The maintainer's comment provides a log output showing that Pipenv successfully recognizes the platform-specific marker and skips the dependency as expected.
  • However, the error still occurs during the locking process, indicating a problem in how the package specifier is processed.

3. Proposed Resolution:

The problem is likely caused by the order of operations when handling the version and markers keys within the python-magic-bin package definition. Here's a breakdown of the likely scenario:

  1. Version Parsing: Pipenv initially parses the version key, encountering the "==0.4." scheme. It correctly interprets this as a wildcard version, potentially replacing it with "".
  2. Marker Handling: Next, the markers key is processed. However, at this stage, the version key may have already been modified, leading to an incomplete package specifier.
  3. Requirement Construction: During the construction of the InstallRequirement object, the incomplete specifier "python-magic-bin0.4.*" (without the "==") is used, resulting in the InvalidRequirement error.

To resolve this, Pipenv should handle the version and markers keys simultaneously, ensuring that the full package specifier, including the version scheme and markers, is used during InstallRequirement construction.

4. Code Snippet (pipenv/utils/dependencies.py):

def install_req_from_pipfile(name, pipfile):
	  # ... (Existing Code) ...

	  # Combine version and markers before constructing the requirement string
	  version = pipfile.get("version", "").replace("==", "")
	  markers = pipfile.get("markers", "")
	  req_str = f"{name}{extras_str}{version}{markers}"

	  install_req, _ = expansive_install_req_from_line(
	      req_str,
	      # ... (Existing Arguments) ...
	  )

	  # ... (Rest of the code) ...

5. Additional Steps:

  • Unit Tests: Introduce test cases specifically targeting the combination of versioning schemes and platform-specific markers to prevent regressions.
  • Investigate Underlying Parser: It's worth investigating if the PyParsing library used for requirement parsing could handle this situation more gracefully, potentially by preserving the original version specifier even when wildcards are used.
  • Improve Error Messages: The current error message is cryptic. Improve it to clearly indicate the source of the error, mentioning the missing "==" operator.

This fix should ensure that Pipenv correctly parses and processes package definitions with both versioning schemes and platform-specific markers, leading to a more robust locking process.

matteius avatar Oct 18 '24 21:10 matteius

@OliverFarren When you get a chance, could you check this issue report against https://github.com/pypa/pipenv/pull/6276

Its a larger Pr than just this issue, but since I put a ton of work into refactoring the install routine methods and ensuring we actually don't do full lock resolution when installing packages, but I included a possible refactor based on the above analysis re-run through Claude, so its possible it won't work, too. The larger refactor was done by me, the human.

matteius avatar Oct 19 '24 22:10 matteius

@OliverFarren When you get a chance, could you check this issue report against #6276

Its a larger Pr than just this issue, but since I put a ton of work into refactoring the install routine methods and ensuring we actually don't do full lock resolution when installing packages, but I included a possible refactor based on the above analysis re-run through Claude, so its possible it won't work, too. The larger refactor was done by me, the human.

Hey @matteius I had a look at the PR and it possibly addresses this issue? Looking at the verbose output stack trace above

  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 1097, in install_req_from_pipfile
    install_req, _ = expansive_install_req_from_line(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 996, in expansive_install_req_from_line
    parts = parse_req_from_line(pip_line, line_source)

I can see how this addition of the PR may be a positive change: https://github.com/pypa/pipenv/blob/c16303083a020773ac94cf9a6ad8325704a8e82b/pipenv/utils/dependencies.py#L1135-L1140

Once the PR is merged and there's a new release, happy to update and try and reproduce

OliverFarren avatar Oct 21 '24 10:10 OliverFarren