pipenv --selective-upgrade does not update hashes in Pipfile.lock

Issue description

If I update a package with --selective-upgrade, I can not install the update with sync.

Steps to replicate

pipenv install 'django==2.2.18' pipenv install --selective-upgrade 'django==2.2.19' pipenv --rm pipenv sync

Expected result

pipenv sync should install all packages from Pipfile.lock.

Actual result

$ pipenv sync
Creating a virtualenv for this project...
Pipfile: /home/mogoh/temp/Pipfile
Using /usr/bin/python3.8 (3.8.6) to create virtualenv...
⠹ Creating virtual environment...created virtual environment CPython3.8.6.final.0-64 in 114ms
  creator CPython3Posix(dest=/home/mogoh/.local/share/virtualenvs/temp-O2KvBR8F, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mogoh/.local/share/virtualenv)
    added seed packages: pip==20.2.4, pkg_resources==0.0.0, setuptools==50.3.2, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
✔ Successfully created virtual environment! 
Virtualenv location: /home/mogoh/.virtualenvs/temp-O2KvBR8F
Installing dependencies from Pipfile.lock (d1ee3e)...
An error occurred while installing django==2.2.19 --hash=sha256:c9c994f5e0a032cbd45089798b52e4080f4dea7241c58e3e0636c54146480bb4 --hash=sha256:0eaca08f236bf502a9773e53623f766cc3ceee6453cc41e6de1c8b80f07d2364! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 3/3 — 00:00:00
Installing initially failed dependencies...
[InstallError]:   File "/home/mogoh/.local/lib/python3.8/site-packages/pipenv/cli/command.py", line 684, in sync
[InstallError]:       retcode = do_sync(
[InstallError]:   File "/home/mogoh/.local/lib/python3.8/site-packages/pipenv/core.py", line 2884, in do_sync
[InstallError]:       do_init(
[InstallError]:   File "/home/mogoh/.local/lib/python3.8/site-packages/pipenv/core.py", line 1304, in do_init
[InstallError]:       do_install_dependencies(
[InstallError]:   File "/home/mogoh/.local/lib/python3.8/site-packages/pipenv/core.py", line 899, in do_install_dependencies
[InstallError]:       batch_install(
[InstallError]:   File "/home/mogoh/.local/lib/python3.8/site-packages/pipenv/core.py", line 796, in batch_install
[InstallError]:       _cleanup_procs(procs, failed_deps_queue, retry=retry)
[InstallError]:   File "/home/mogoh/.local/lib/python3.8/site-packages/pipenv/core.py", line 703, in _cleanup_procs
[InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: Collecting django==2.2.19
[pipenv.exceptions.InstallError]:   Using cached Django-2.2.19-py3-none-any.whl (7.5 MB)
[pipenv.exceptions.InstallError]: ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
[pipenv.exceptions.InstallError]:     django==2.2.19 from https://files.pythonhosted.org/packages/1a/ce/846a9dbf536991be0004f5ae414520c3a64eaa167d09e51d75ab410c45e8/Django-2.2.19-py3-none-any.whl#sha256=e319a7164d6d30cb177b3fd74d02c52f1185c37304057bb76d74047889c605d9 (from -r /tmp/pipenv-bdmyjhfi-requirements/pipenv-rkmf8g46-requirement.txt (line 1)):
[pipenv.exceptions.InstallError]:         Expected sha256 0eaca08f236bf502a9773e53623f766cc3ceee6453cc41e6de1c8b80f07d2364
[pipenv.exceptions.InstallError]:         Expected     or c9c994f5e0a032cbd45089798b52e4080f4dea7241c58e3e0636c54146480bb4
[pipenv.exceptions.InstallError]:              Got        e319a7164d6d30cb177b3fd74d02c52f1185c37304057bb76d74047889c605d9
ERROR: Couldn't install package: django
 Package installation failed...
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

$ pipenv --support

Pipenv version: '2020.11.15'

Pipenv location: '/home/mogoh/.local/lib/python3.8/site-packages/pipenv'

Python location: '/usr/bin/python3'

Python installations found:

3.8.6: /usr/bin/python3.8
3.8.6: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.8.0-43-generic',
 'platform_system': 'Linux',
 'platform_version': '#49-Ubuntu SMP Fri Feb 5 03:01:28 UTC 2021',
 'python_full_version': '3.8.6',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

SHELL
SESSION_MANAGER
WINDOWID
QT_ACCESSIBILITY
KDED_STARTED_BY_KDEINIT
COLORTERM
XDG_CONFIG_DIRS
XDG_SESSION_PATH
NVM_INC
GTK_IM_MODULE
LANGUAGE
QT4_IM_MODULE
MANDATORY_PATH
JAVA_HOME
SSH_AUTH_SOCK
SHELL_SESSION_ID
XMODIFIERS
DESKTOP_SESSION
SSH_AGENT_PID
GTK_RC_FILES
XCURSOR_SIZE
XDG_SEAT
PWD
XDG_SESSION_DESKTOP
LOGNAME
XDG_SESSION_TYPE
GPG_AGENT_INFO
XAUTHORITY
VIRTUALENVWRAPPER_SCRIPT
GTK2_RC_FILES
HOME
LANG
LS_COLORS
XDG_CURRENT_DESKTOP
KONSOLE_DBUS_SERVICE
VIRTUALENVWRAPPER_WORKON_CD
KONSOLE_DBUS_SESSION
DENO_INSTALL
PROFILEHOME
XDG_SEAT_PATH
KONSOLE_VERSION
CLUTTER_IM_MODULE
KDE_SESSION_UID
NVM_DIR
WORKON_HOME
LESSCLOSE
XDG_SESSION_CLASS
PYTHONPATH
TERM
DEFAULTS_PATH
LESSOPEN
USER
COLORFGBG
KDE_SESSION_VERSION
PAM_KWALLET5_LOGIN
VIRTUALENVWRAPPER_PROJECT_FILENAME
DISPLAY
SHLVL
NVM_CD_FLAGS
QT_IM_MODULE
XDG_VTNR
XDG_SESSION_ID
XDG_RUNTIME_DIR
ELECTRON_TRASH
QT_AUTO_SCREEN_SCALE_FACTOR
XCURSOR_THEME
XDG_DATA_DIRS
KDE_FULL_SESSION
PATH
VIRTUALENVWRAPPER_HOOK_DIR
DBUS_SESSION_BUS_ADDRESS
KDE_APPLICATIONS_AS_SCOPE
NVM_BIN
OLDPWD
KONSOLE_DBUS_WINDOW
_
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /home/mogoh/.yarn/bin:/home/mogoh/.deno/bin:/home/mogoh/.nvm/versions/node/v15.8.0/bin:/home/mogoh/.gem/ruby/2.5.0/bin:/home/mogoh/.cargo/bin:/home/mogoh/.local/bin:/home/mogoh/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL: /bin/bash
LANG: de_DE.UTF-8
PWD: /home/mogoh/temp

Contents of Pipfile ('/home/mogoh/temp/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
django = "==2.2.19"

[dev-packages]

[requires]
python_version = "3.8"

Contents of Pipfile.lock ('/home/mogoh/temp/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "931594d6e3f80b678b899b9419026dedb97c8880fe21c17c4297f8ed2ad1ee3e"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "django": {
            "hashes": [
                "sha256:0eaca08f236bf502a9773e53623f766cc3ceee6453cc41e6de1c8b80f07d2364",
                "sha256:c9c994f5e0a032cbd45089798b52e4080f4dea7241c58e3e0636c54146480bb4"
            ],
            "index": "pypi",
            "version": "==2.2.19"
        },
        "pytz": {
            "hashes": [
                "sha256:83a4a90894bf38e243cf052c8b58f381bfe9a7a483f6a9cab140bc7f702ac4da",
                "sha256:eb10ce3e7736052ed3623d49975ce333bcd712c7bb19a58b9e2089d4057d0798"
            ],
            "version": "==2021.1"
        },
        "sqlparse": {
            "hashes": [
                "sha256:017cde379adbd6a1f15a61873f43e8274179378e95ef3fede90b5aa64d304ed0",
                "sha256:0f91fd2e829c44362cbcfab3e9ae12e22badaa8a29ad5ff599f9ec109f0454e8"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==0.4.1"
        }
    },
    "develop": {}
}

Feb 19 '21 17:02 mogoh

I see this behavior too, with simply a --keep-outdated -- it breaks the Pipfile.lock because it does not update the hashes for the package you update.

$ pipenv --support

Pipenv version: '2021.11.23'

Pipenv location: '/home/jsnow/.local/lib/python3.9/site-packages/pipenv'

Python location: '/usr/bin/python3'

Python installations found:

3.9.7: /usr/bin/python3-bpython
3.9.7: /usr/bin/python3
3.9.7: /usr/bin/python3.9
3.9.7: /usr/bin/python
3.8.12: /usr/bin/python3.8
3.7.12: /usr/bin/python3.7
3.7.12: /usr/bin/python3.7m
3.6.15: /usr/bin/python3.6
3.6.15: /usr/bin/python3.6m
3.6.9: /usr/bin/pypy3.6
3.6.9: /usr/bin/pypy3
3.5.10: /usr/bin/python3.5
3.5.10: /usr/bin/python3.5m
2.7.18: /usr/bin/python2
2.7.18: /usr/bin/python2.7
2.7.13: /usr/bin/pypy
2.7.13: /usr/bin/pypy2.7
2.7.13: /usr/bin/pypy2

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.7',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.14.14-200.fc34.x86_64',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Oct 20 16:15:12 UTC 2021',
 'python_full_version': '3.9.7',
 'python_version': '3.9',
 'sys_platform': 'linux'}

Jan 19 '22 17:01 jnsnow

@mogoh and @jnsnow Have you tried this with pipenv==2022.1.8?

Mar 13 '22 10:03 matteius

I got still the same problem with pipenv version 2022.1.8.

$ pipenv sync
Creating a virtualenv for this project...
Pipfile: /home/mogoh/temp/pipenvtest/Pipfile
Using /usr/bin/python3.9 (3.9.7) to create virtualenv...
⠹ Creating virtual environment...created virtual environment CPython3.9.7.final.0-64 in 165ms
  creator CPython3Posix(dest=/home/mogoh/.local/share/virtualenvs/pipenvtest-LdIeJRsO, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mogoh/.local/share/virtualenv)
    added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /home/mogoh/.virtualenvs/pipenvtest-LdIeJRsO
Installing dependencies from Pipfile.lock (b6ef97)...
An error occurred while installing django==2.2.19 --hash=sha256:c9c994f5e0a032cbd45089798b52e4080f4dea7241c58e3e0636c54146480bb4 --hash=sha256:0eaca08f236bf502a9773e53623f766cc3ceee6453cc41e6de1c8b80f07d2364! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 3/3 — 00:00:00
Installing initially failed dependencies...
[pipenv.exceptions.InstallError]: Collecting django==2.2.19
[pipenv.exceptions.InstallError]:   Using cached Django-2.2.19-py3-none-any.whl (7.5 MB)
[pipenv.exceptions.InstallError]: ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
[pipenv.exceptions.InstallError]:     django==2.2.19 from https://files.pythonhosted.org/packages/1a/ce/846a9dbf536991be0004f5ae414520c3a64eaa167d09e51d75ab410c45e8/Django-2.2.19-py3-none-any.whl#sha256=e319a7164d6d30cb177b3fd74d02c52f1185c37304057bb76d74047889c605d9 (from -r /tmp/pipenv-_us227jm-requirements/pipenv-jk4w0918-requirement.txt (line 1)):
[pipenv.exceptions.InstallError]:         Expected sha256 0eaca08f236bf502a9773e53623f766cc3ceee6453cc41e6de1c8b80f07d2364
[pipenv.exceptions.InstallError]:         Expected     or c9c994f5e0a032cbd45089798b52e4080f4dea7241c58e3e0636c54146480bb4
[pipenv.exceptions.InstallError]:              Got        e319a7164d6d30cb177b3fd74d02c52f1185c37304057bb76d74047889c605d9
ERROR: Couldn't install package: django
 Package installation failed...
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

Mar 14 '22 12:03 mogoh

@mogoh and @jnsnow Have you tried this with pipenv==2022.1.8?

I'll check again on Monday. I see you are proposing removing selective-upgrade, but I was seeing problems with --keep-outdated, too. Let me try to give you a more cohesive bug report using the flag that it looks like you'd rather keep.

May 14 '22 01:05 jnsnow

I was seeing a similar behaviour to what @jnsnow reported (i.e with --keep-outdated), and yes I did try with pipenv==2022.1.8 but it made no difference.

Since there's no fix as of yet, my workaround was simply just deleting from the Pipfile.lock the block(s) of the package(s) that you actually wanna update. Seems when you run pipenv lock --keep-outdated again, it regenerates them with their new hashes.

Disclaimer: I did not delete the blocks of their transitive dependencies in the Pipfile.lock, and they weren't changed after I ran pipenv lock --keep-outdated again, so I'm not sure if they weren't changed because pipenv saw no need to update them as it realized they were still compatible, or if pipenv didn't even try to update them and I was just lucky that they were still compatible.

Anyway, hope this helps.

Aug 05 '22 22:08 Louai-Abdelsalam

Sorry I haven't gotten back around to this yet, but this is really useful feedback. I am not sure that dropping those flags is going to make everyone happy, but I do know that the implementations of them are flawed, and its essentially pretty challenging to support such a feature. I personally think correct usage should be to set the specifiers in Pipfile in a way that respects what packages you require, and then trust the resolver during lock and install (which locks) to updating to the latest versions of packages that you have specified. The problem with doing anything different than that is there is potential for dependencies collide and create incompatible lock files. Dropping support for these features has felt like the path forward to this for me, but I could be convinced if I saw an implementation that made sense for --keep-outdated or --selective-upgrade -- perhaps it involves use of constraint files.

Aug 15 '22 13:08 matteius

As long as dropping support does not entail removing the feature altogether. It is a pretty useful feature honestly when all your package versions are pinned, so I'd rather it be kept unsupported, than entirely removed.

Anyway, I'll leave the contribution guidelines here in case anyone decides to work on it: https://pipenv.pypa.io/en/latest/dev/contributing/

Aug 18 '22 20:08 Louai-Abdelsalam

My team has run into this a few times now when trying to upgrade only one package using pipenv install --selective-upgrade <package>==<version>. Our workaround has been similar to @Louai-Abdelsalam's fix, but we instead uninstall the package and then re-install it with the new version.

This issue has caused us enough problems that we are considering moving away from pipenv.

Nov 18 '22 21:11 yodaldevoid

@yodaldevoid I believe the issue is that --selective-upgrade is working as intended, but that is hard to evaluate because it was never documented (not in the docs). I generally advise people not to use --selective-upgrade and to re-lock all dependencies. Your Pipfile should have reasonable specifiers for what you expect so that re-locking everything should be a non-issue (when proper specifiers are set for your project). I've advocated for getting rid of --selective-upgrade but some folks want it kept around for some reason, even though it can create lock files that are inconsistent with the actual dependencies, which is why Pip would fail to install them. Same goes for --keep-outdated.

Nov 18 '22 21:11 matteius

@matteius If I understand correctly, you are suggesting that we pin dependencies in our Pipfile rather than relying on the pined versions in Pipfile.lock in conjunction with --selective-upgrade to only upgrade selective dependencies, is that correct?

Nov 18 '22 21:11 yodaldevoid

@yodaldevoid I am suggesting not to Pin all of your dependencies in your Pipfile, but rather figure out which ones are the reason why you are using --selective-upgrade in the first place and pick reasonable specifiers for those. For example, let's say you know you need Django >=3 .* but < 4 -- you can specify this in your Pipfile, but not worry about all of the downstream dependencies that are going to be pulled in by Django. You wills till generate a lock file that has all of the exact specifiers used for the sync/instatll phase, but you would unravel what is causing you to use --selective-upgrade in the first place.

The only reason I can see people are using this and keep-outdated, is they have things in the past that upgraded when locking the whole package group, that they feared are going to get changed, but this is due to not understanding what those things are and creating reasonably specifiers in the Pipfile.

Another possible work around is move sensitive dependencies you don't want upgraded into their own package category and when you lock you exclude that category from being locked, but I still think Pipfile specifiers for top level dependencies make the most sense.

Nov 18 '22 21:11 matteius

pipenv update and pipenv upgrade were re-designed to replace --keep-outdated and --selective-upgrade and this work has been completed already.

Aug 26 '23 05:08 matteius

pipenv pipenv copied to clipboard

--selective-upgrade does not update hashes in Pipfile.lock

Issue description

Steps to replicate

Expected result

Actual result

pipenv
pipenv copied to clipboard