pipenv Pinned version in dev-packages ignored when used by main packages

Hi, this may be expected behaviour but I wanted to double check. Couldn't find a clear answer in the docs or in an issue.

Issue description

When I pin a dependency in my dev-packages, I would expect pipenv to honour that and either:

Install that version as part of the dev packages
Or fail with a resolution failure if there is another package trying to install a different version from the one I specified.

Expected result

In my example below, if I pin packaging to v21.3 in my dev-packages, I would expect to either see that version or get resolution failure.

Actual result

In my example below, given that apispec requires any version of packaging, it is installing v22.0 in both packages and dev-packages. I wouldn't expect to see this version in dev-packages, instead I would expect to see v21.3 or resolution failure.

Steps to replicate

Provided pipenv --support output below.

Thanks in advance.

Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.

$ pipenv --support

Pipenv version: '2022.11.30'

Pipenv location: '/home/REDACTED/lib/python3.9/site-packages/pipenv'

Python location: '/home/REDACTED/bin/python'

OS Name: 'posix'

User pip version: '22.3'

user Python installations found:

3.9.16: /home/REDACTED/bin/python3.9
3.9.16: /home/REDACTED/bin/python3
3.9.16: /home/REDACTED/bin/python
3.9.16: /home/REDACTED/bin/python3.9
3.9.16: /home/REDACTED/bin/python3
3.9.16: /home/REDACTED/bin/python
3.9.16: /usr/bin/python3.9
3.9.16: /bin/python3.9
3.8.10: /usr/bin/python3
3.8.10: /usr/bin/python3.8
3.8.10: /bin/python3
3.8.10: /bin/python3.8
3.7.16: /usr/bin/python3.7m
3.7.16: /usr/bin/python3.7
3.7.16: /bin/python3.7m
3.7.16: /bin/python3.7
2.7.18: /usr/bin/python2.7
2.7.18: /usr/bin/python2
2.7.18: /bin/python2.7
2.7.18: /bin/python2

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.16',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.15.0-56-generic',
 'platform_system': 'Linux',
 'platform_version': '#62~20.04.1-Ubuntu SMP Tue Nov 22 21:24:20 UTC 2022',
 'python_full_version': '3.9.16',
 'python_version': '3.9',
 'sys_platform': 'linux'}

System environment variables:

SHELL
SESSION_MANAGER
QT_ACCESSIBILITY
NVM_RC_VERSION
SNAP_REVISION
XDG_CONFIG_DIRS
XDG_MENU_PREFIX
GNOME_DESKTOP_SESSION_ID
SNAP_REAL_HOME
TERMINAL_EMULATOR
SNAP_USER_COMMON
LANGUAGE
GNOME_SHELL_SESSION_MODE
SSH_AUTH_SOCK
TERM_SESSION_ID
SNAP_INSTANCE_KEY
XMODIFIERS
DESKTOP_SESSION
SSH_AGENT_PID
BAMF_DESKTOP_FILE_HINT
GTK_MODULES
PWD
XDG_SESSION_DESKTOP
LOGNAME
XDG_SESSION_TYPE
GPG_AGENT_INFO
XAUTHORITY
DESKTOP_STARTUP_ID
SNAP_CONTEXT
GJS_DEBUG_TOPICS
WINDOWPATH
HOME
USERNAME
IM_CONFIG_PHASE
LANG
LS_COLORS
XDG_CURRENT_DESKTOP
VIRTUAL_ENV
SNAP_ARCH
SNAP_INSTANCE_NAME
SNAP_USER_DATA
INVOCATION_ID
MANAGERPID
SNAP_REEXEC
GJS_DEBUG_OUTPUT
NVM_DIR
LESSCLOSE
XDG_SESSION_CLASS
TERM
AWS_EIO_CD_PROFILE
LESSOPEN
USER
SNAP
SNAP_COMMON
SNAP_VERSION
DISPLAY
SHLVL
NVM_CD_FLAGS
SNAP_LIBRARY_PATH
SNAP_COOKIE
QT_IM_MODULE
SNAP_DATA
XDG_RUNTIME_DIR
PS1
SNAP_NAME
JOURNAL_STREAM
XDG_DATA_DIRS
PATH
GDMSESSION
DBUS_SESSION_BUS_ADDRESS
GIO_LAUNCHED_DESKTOP_FILE_PID
GIO_LAUNCHED_DESKTOP_FILE
_
PIP_DISABLE_PIP_VERSION_CHECK
PIP_PYTHON_PATH
PYTHONDONTWRITEBYTECODE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /home/REDACTED/bin:/home/REDACTED/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL: /bin/bash
LANG: en_CA.UTF-8
PWD: /home/REDACTED
VIRTUAL_ENV: /home/REDACTED

Contents of Pipfile ('/home/REDACTED/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
apispec = "*"

[requires]
python_version = "3.9"

[pipenv]
allow_prereleases = true

[dev-packages]
packaging = "==21.3"

Contents of Pipfile.lock ('/home/REDACTED/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "46f6381ed3ae611fdfc008749d0e138dacb07e741b0b7888ecc136adbbfad371"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.9"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "apispec": {
            "hashes": [
                "sha256:d97f0ae9c65133185b9ed9c5be1a434eb85627dfa33c4c53cabda122256c1b67",
                "sha256:e76d80b739edef4be213092a6384ad7fd933ba7d64f6d5a0aff8d4da1bef7887"
            ],
            "index": "pypi",
            "version": "==6.0.2"
        },
        "packaging": {
            "hashes": [
                "sha256:2198ec20bd4c017b8f9717e00f0c8714076fc2fd93816750ab48e2c41de2cfd3",
                "sha256:957e2148ba0e1a3b282772e791ef1d8083648bc131c8ab0c1feba110ce1146c3"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==22.0"
        }
    },
    "develop": {
        "packaging": {
            "hashes": [
                "sha256:2198ec20bd4c017b8f9717e00f0c8714076fc2fd93816750ab48e2c41de2cfd3",
                "sha256:957e2148ba0e1a3b282772e791ef1d8083648bc131c8ab0c1feba110ce1146c3"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==22.0"
        },
        "pyparsing": {
            "hashes": [
                "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
                "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
            ],
            "markers": "python_full_version >= '3.6.8'",
            "version": "==3.0.9"
        }
    }
}

Dec 08 '22 17:12 juanitosvq

@juanitosvq Yeah this is expected behavior; there exists a compromise between the desire to not have something like django-debug-toolbar upgrade the version of Django in dev that is pinned in packages. During the addition of named package categories I decided to continue having the default or packages group continue to constrain (this is pip terminology, we use a constraint file for this) the other package groups.

For users that do not want this behavior I am considering a global pipenv option to not constrain from default, but right now the recommendation is to use named package categories. For your example you could have a prod group that included the apispec version you wanted, or not use default/packages group at all and nothing would be constrained this way.

I guess if there is enough interest in an option in the Pipfile that does not constrain when locking, we can do it. Then the support flow becomes --> default is constrained (issue reported) --> you switch to use no constraints (different issue reported) --> Compromise and keep things you want constrained in default packages group, and things you don't want constrained defined in other named categories.

The documentation could/should be improved, but there is a note about it here: https://pipenv.pypa.io/en/latest/basics/#specifying-package-categories

Dec 08 '22 20:12 matteius

Thanks for the prompt and detailed response, @matteius . I wasn't aware of the named package categories feature, I think that may help us sorting out the issue we are having, thanks!

Dec 09 '22 05:12 juanitosvq

For now, the way I got succeed to get expected behavior with pipenv==2023.2.4 is:

Succeed example

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

- [packages]
+ [prod-packages]
apispec = "==6.0.2"

- [dev-packages]
+ [packages]
packaging = "==21.3"
colorama = "==0.4.6"

Then:

$ pipenv lock

Creating a virtualenv for this project...
Pipfile: /workspace/Pipfile
Using /usr/local/bin/python (3.11.1) to create virtualenv...
⠙ Creating virtual environment...created virtual environment CPython3.11.1.final.0-64 in 120ms
  creator CPython3Posix(dest=/workspace/.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.3.1, setuptools==66.0.0, wheel==0.38.4
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /workspace/.venv
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Locking [prod-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (11b641784365aea8803e3ee8c4be5a3a43e6691a28cd2b0b0bfeac06a83318f4)!

Pipfile.lock

{
    "_meta": {
        "hash": {
            "sha256": "11b641784365aea8803e3ee8c4be5a3a43e6691a28cd2b0b0bfeac06a83318f4"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "colorama": {
            "hashes": [
                "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44",
                "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"
            ],
            "index": "pypi",
            "version": "==0.4.6"
        },
        "packaging": {
            "hashes": [
                "sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb",
                "sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522"
            ],
            "index": "pypi",
            "version": "==21.3"
        },
        "pyparsing": {
            "hashes": [
                "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
                "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
            ],
            "markers": "python_full_version >= '3.6.8'",
            "version": "==3.0.9"
        }
    },
    "develop": {},
    "prod-packages": {
        "apispec": {
            "hashes": [
                "sha256:d97f0ae9c65133185b9ed9c5be1a434eb85627dfa33c4c53cabda122256c1b67",
                "sha256:e76d80b739edef4be213092a6384ad7fd933ba7d64f6d5a0aff8d4da1bef7887"
            ],
            "index": "pypi",
            "version": "==6.0.2"
        },
        "packaging": {
            "hashes": [
                "sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb",
                "sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522"
            ],
            "index": "pypi",
            "version": "==21.3"
        },
        "pyparsing": {
            "hashes": [
                "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
                "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
            ],
            "markers": "python_full_version >= '3.6.8'",
            "version": "==3.0.9"
        }
    }
}

To create development environment:

pipenv sync --categories='packages prod-packages'

Result:

$ pipenv run pip list
Package    Version
---------- -------
apispec    6.0.2
colorama   0.4.6
packaging  21.3
pip        22.3.1
pyparsing  3.0.9
setuptools 66.0.0
wheel      0.38.4

The command to deploy into production:

pipenv install --deploy --categories=prod-packages

Result:

$ pipenv run pip list
Package             Version
------------------- -------
Package    Version
---------- -------
apispec    6.0.2
packaging  21.3
pip        22.3.1
pyparsing  3.0.9
setuptools 66.0.0
wheel      0.38.4

Failed example

The following is failed:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

- [packages]
+ [prod-packages]
apispec = "==6.0.2"

[dev-packages]
packaging = "==21.3"
colorama = "==0.4.6"

The locked packaging versions were different between development and production:

Pipfile.lock

{
    "_meta": {
        "hash": {
            "sha256": "23f267f4e6a9557b26caadaf4a4574299298594ec6690a5d0b64fe39ef568023"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {
        "colorama": {
            "hashes": [
                "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44",
                "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"
            ],
            "index": "pypi",
            "version": "==0.4.6"
        },
        "packaging": {
            "hashes": [
                "sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb",
                "sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522"
            ],
            "index": "pypi",
            "version": "==21.3"
        },
        "pyparsing": {
            "hashes": [
                "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
                "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
            ],
            "markers": "python_full_version >= '3.6.8'",
            "version": "==3.0.9"
        }
    },
    "prod-packages": {
        "apispec": {
            "hashes": [
                "sha256:d97f0ae9c65133185b9ed9c5be1a434eb85627dfa33c4c53cabda122256c1b67",
                "sha256:e76d80b739edef4be213092a6384ad7fd933ba7d64f6d5a0aff8d4da1bef7887"
            ],
            "index": "pypi",
            "version": "==6.0.2"
        },
        "packaging": {
            "hashes": [
                "sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2",
                "sha256:b6ad297f8907de0fa2fe1ccbd26fdaf387f5f47c7275fedf8cce89f99446cf97"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==23.0"
        }
    }
}

Feb 17 '23 11:02 yukihiko-shinoda

Can't it be thought that many people who use dependency management tools do so because they want to ensure same packages in test and production environments? And aren't there so many users who don't know this specification change of behavior?

https://github.com/pypa/pipenv/pull/5366

https://github.com/pypa/pipenv/pull/5366/files#diff-ef852c4ac364f946819f765a6bc26f04f1b0968f31fc512949a60fa2ab0685e8R1027

Behavior of lock function in representative dependency management tools

I checked followings:

PHP Composer
Ruby Bundler
Python Poetry
Python PDM
Node Npm
Node Yarn
Golang Go Modules
Java Maven
Java Gradle

Almost dependency management tools lock by both default category and dev-package. If dev-package category locks lower version than default category, locked version is adjusted to lower version. Only Gradle prioritize default category than dev-package category, however it locks into only 1 version of each package which is common with all categories, and it also supports function to pin transitive dependencies.

Feb 18 '23 18:02 yukihiko-shinoda

I am wondering if this could be rechecked on 2024.3.0?

Oct 30 '24 11:10 matteius

pipenv pipenv copied to clipboard

Pinned version in dev-packages ignored when used by main packages

Issue description

Expected result

Actual result

Steps to replicate

Succeed example

Failed example

Behavior of lock function in representative dependency management tools

pipenv
pipenv copied to clipboard