pipenv Install package with requirement specifiers doesn't write dependencies

Issue description

When installing a package with a simple requirement specifier, pipenv.lock doesn't show any dependencies if locking was made with a platform which doesn't match the constraint.

It is working with pipenv 2018.11.26 on windows.

Pipfile.lock with pipenv 2018.11.26 on both Windows 10 & Ubuntu 20.04


{
    "_meta": {
        "hash": {
            "sha256": "7624ac65f711c91e061df6c8be22fb968565da45132a67f6dd553f97c531ad95"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "pyasn1": {
            "hashes": [
                "sha256:39c7e2ec30515947ff4e87fb6f456dfc6e84857d34be479c9d4a4ba4bf46aa5d",
                "sha256:aef77c9fb94a3ac588e87841208bdec464471d9871bd5050a287cc9a475cd0ba"
            ],
            "version": "==0.4.8"
        },
        "pyasn1-modules": {
            "hashes": [
                "sha256:905f84c712230b2c592c19470d3ca8d552de726050d1d1716282a1f6146be65e",
                "sha256:a50b808ffeb97cb3601dd25981f6b016cbb3d31fbf57a8b8a87428e6158d0c74"
            ],
            "version": "==0.2.8"
        },
        "python-ldap": {
            "hashes": [
                "sha256:4711cacf013e298754abd70058ccc995758177fb425f1c2d30e71adfc1d00aa5"
            ],
            "index": "pypi",
            "markers": "platform_system == 'Linux'",
            "version": "==3.3.1"
        }
    },
    "develop": {}
}

Expected result

Lock file with specified package & dependencies like doing it on the specified platform. At least like pipenv 2018.11.26

This is the generated lock file on WSL.

{
    "_meta": {
        "hash": {
            "sha256": "7624ac65f711c91e061df6c8be22fb968565da45132a67f6dd553f97c531ad95"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "pyasn1": {
            "hashes": [
                "sha256:014c0e9976956a08139dc0712ae195324a75e142284d5f87f1a87ee1b068a359",
                "sha256:03840c999ba71680a131cfaee6fab142e1ed9bbd9c693e285cc6aca0d555e576",
                "sha256:0458773cfe65b153891ac249bcf1b5f8f320b7c2ce462151f8fa74de8934becf",
                "sha256:08c3c53b75eaa48d71cf8c710312316392ed40899cb34710d092e96745a358b7",
                "sha256:39c7e2ec30515947ff4e87fb6f456dfc6e84857d34be479c9d4a4ba4bf46aa5d",
                "sha256:5c9414dcfede6e441f7e8f81b43b34e834731003427e5b09e4e00e3172a10f00",
                "sha256:6e7545f1a61025a4e58bb336952c5061697da694db1cae97b116e9c46abcf7c8",
                "sha256:78fa6da68ed2727915c4767bb386ab32cdba863caa7dbe473eaae45f9959da86",
                "sha256:7ab8a544af125fb704feadb008c99a88805126fb525280b2270bb25cc1d78a12",
                "sha256:99fcc3c8d804d1bc6d9a099921e39d827026409a58f2a720dcdb89374ea0c776",
                "sha256:aef77c9fb94a3ac588e87841208bdec464471d9871bd5050a287cc9a475cd0ba",
                "sha256:e89bf84b5437b532b0803ba5c9a5e054d21fec423a89952a74f87fa2c9b7bce2",
                "sha256:fec3e9d8e36808a28efb59b489e4528c10ad0f480e57dcc32b4de5c9d8c9fdf3"
            ],
            "version": "==0.4.8"
        },
        "pyasn1-modules": {
            "hashes": [
                "sha256:0845a5582f6a02bb3e1bde9ecfc4bfcae6ec3210dd270522fee602365430c3f8",
                "sha256:0fe1b68d1e486a1ed5473f1302bd991c1611d319bba158e98b106ff86e1d7199",
                "sha256:15b7c67fabc7fc240d87fb9aabf999cf82311a6d6fb2c70d00d3d0604878c811",
                "sha256:426edb7a5e8879f1ec54a1864f16b882c2837bfd06eee62f2c982315ee2473ed",
                "sha256:65cebbaffc913f4fe9e4808735c95ea22d7a7775646ab690518c056784bc21b4",
                "sha256:905f84c712230b2c592c19470d3ca8d552de726050d1d1716282a1f6146be65e",
                "sha256:a50b808ffeb97cb3601dd25981f6b016cbb3d31fbf57a8b8a87428e6158d0c74",
                "sha256:a99324196732f53093a84c4369c996713eb8c89d360a496b599fb1a9c47fc3eb",
                "sha256:b80486a6c77252ea3a3e9b1e360bc9cf28eaac41263d173c032581ad2f20fe45",
                "sha256:c29a5e5cc7a3f05926aff34e097e84f8589cd790ce0ed41b67aed6857b26aafd",
                "sha256:cbac4bc38d117f2a49aeedec4407d23e8866ea4ac27ff2cf7fb3e5b570df19e0",
                "sha256:f39edd8c4ecaa4556e989147ebf219227e2cd2e8a43c7e7fcb1f1c18c5fd6a3d",
                "sha256:fe0644d9ab041506b62782e92b06b8c68cca799e1a9636ec398675459e031405"
            ],
            "version": "==0.2.8"
        },
        "python-ldap": {
            "hashes": [
                "sha256:4711cacf013e298754abd70058ccc995758177fb425f1c2d30e71adfc1d00aa5"
            ],
            "index": "pypi",
            "markers": "platform_system == 'Linux'",
            "version": "==3.3.1"
        }
    },
    "develop": {}
}

Actual result

Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
           Building requirements...
Resolving dependencies...
[====] Locking...
                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Generating hashes:
[=== ] Locking...ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Generating hashes:
Success!
Updated Pipfile.lock (31ad95)!

{
    "_meta": {
        "hash": {
            "sha256": "7624ac65f711c91e061df6c8be22fb968565da45132a67f6dd553f97c531ad95"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "python-ldap": {
            "hashes": [
                "sha256:4711cacf013e298754abd70058ccc995758177fb425f1c2d30e71adfc1d00aa5"
            ],
            "markers": "platform_system == 'Linux'",
            "version": "==3.3.1"
        }
    },
    "develop": {}
}

Steps to replicate

Pipfile :

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
python-ldap = {markers = "platform_system=='Linux'", version = "*"}

[requires]
python_version = "3.8"

pipenv lock

$ pipenv --support

Pipenv version: '2020.6.2'

Pipenv location: 'C:\\Users\\Jules\\AppData\\Roaming\\Python\\Python38\\site-packages\\pipenv'

Python location: 'c:\\python38\\python.exe'

Python installations found:

3.8.2: C:\Python38\python.exe
3.7.3: C:\Python37\python.exe
3.6.8: C:\Python36\python.exe
2.7.17: C:\Python27\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.2',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.19041',
 'python_full_version': '3.8.2',
 'python_version': '3.8',
 'sys_platform': 'win32'}

System environment variables:

ALLUSERSPROFILE
APPDATA
APPLICATION_INSIGHTS_NO_DIAGNOSTIC_CHANNEL
CHOCOLATEYINSTALL
CHOCOLATEYLASTPATHUPDATE
COMMONPROGRAMFILES
COMMONPROGRAMFILES(X86)
COMMONPROGRAMW6432
COMPOSE_CONVERT_WINDOWS_PATHS
COMPUTERNAME
COMSPEC
CUDA_PATH
CUDA_PATH_V10_0
CUDA_PATH_V10_1
CUDA_PATH_V9_0
CUDA_PATH_V9_1
DIGITALOCEAN_ACCESS_TOKEN
DIGITALOCEAN_IMAGE
DIGITALOCEAN_REGION
DIGITALOCEAN_SSH_KEY_FINGERPRINT
DRIVERDATA
FPS_BROWSER_APP_PROFILE_STRING
FPS_BROWSER_USER_PROFILE_STRING
HOMEDRIVE
HOMEPATH
INCLUDE
JAVA_HOME
LD_LIBRARY_PATH
LOCALAPPDATA
LOGONSERVER
NUMBER_OF_PROCESSORS
NVCUDASAMPLES10_0_ROOT
NVCUDASAMPLES10_1_ROOT
NVCUDASAMPLES9_0_ROOT
NVCUDASAMPLES9_1_ROOT
NVCUDASAMPLES_ROOT
NVTOOLSEXT_PATH
OCULUSBASE
ONEDRIVE
ONEDRIVECONSUMER
ORIGINAL_XDG_CURRENT_DESKTOP
OS
PATH
PATHEXT
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
PROCESSOR_LEVEL
PROCESSOR_REVISION
PROGRAMDATA
PROGRAMFILES
PROGRAMFILES(X86)
PROGRAMW6432
PROMPT
PSMODULEPATH
PUBLIC
PYCHARM
SESSIONNAME
SYSTEMDRIVE
SYSTEMROOT
TEMP
TMP
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERNAME
USERPROFILE
VS140COMNTOOLS
WINDIR
TERM_PROGRAM
TERM_PROGRAM_VERSION
LANG
COLORTERM
VSCODE_GIT_IPC_HANDLE
GIT_ASKPASS
VSCODE_GIT_ASKPASS_NODE
VSCODE_GIT_ASKPASS_MAIN
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

PipenvÔÇôspecific environment variables:

DebugÔÇôspecific environment variables:

PATH: C:\openssl;C:\Program Files\Amazon Corretto\jdk11.0.6_10\bin;C:\Python38\Scripts\;C:\Python38\;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\libnvvp;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Python37\Scripts;C:\Python37\;C:\Python36\Scripts\;C:\Python36\;C:\Program Files\Oculus\Support\oculus-runtime;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\libnvvp;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\ProgramData\chocolatey\bin;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\OpenVPN\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\Program Files (x86)\Gpg4win\..\GnuPG\bin;C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;C:\Program Files\dotnet\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\nodejs\;C:\Program Files (x86)\Yarn\bin\;C:\Program Files\Amazon\AWSCLI\bin\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Users\Jules\.poetry\bin;C:\Users\Jules\.cargo\bin;C:\Users\Jules\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Google\Chrome\Application;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin;C:\Users\Jules\AppData\Roaming\Mist;C:\Users\Jules\AppData\Roaming\Ethereum Wallet\binaries\Geth\unpacked;C:\Program Files\Git\bin;C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\bin\HostX86\x86;C:\Miner\ethminer\bin;C:\Program Files\nodejs;D:\06.Dev-Tools\Qt\Qt5.11.2\5.11.2\msvc2017_64\bin;C:\Users\Jules\.dotnet\tools;C:\Program Files\JetBrains\PyCharm 2019.1.4\bin;;C:\Users\Jules\Downloads;C:\Program Files\ffmpeg-4.2.3-win64-static\bin;C:\Program Files (x86)\phantomjs-2.1.1-windows\bin;C:\Program Files (x86)\balena-cli;C:\Users\Jules\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Jules\AppData\Local\Microsoft\WindowsApps;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\extras\CUPTI\libx64;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\include;C:\tools\cuda\bin;C:\Users\Jules\AppData\Roaming\npm;C:\Users\Jules\AppData\Local\Yarn\bin;C:\Program Files\Terraform;C:\Users\Jules\bin;C:\Gradle\gradle-6.3\bin;C:\Program Files (x86)\Graphviz2.38\bin;C:\Users\Jules\AppData\Local\Pandoc\;C:\Users\Jules\AppData\Roaming\Python\Python38\Scripts;
LANG: fr_FR.UTF-8

Contents of Pipfile ('C:\Users\Jules\Code\test-pipenv\Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
python-ldap = {markers = "platform_system=='Linux'", version = "*"}

[requires]
python_version = "3.8"

Contents of Pipfile.lock ('C:\Users\Jules\Code\test-pipenv\Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "7624ac65f711c91e061df6c8be22fb968565da45132a67f6dd553f97c531ad95"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "python-ldap": {
            "hashes": [
                "sha256:4711cacf013e298754abd70058ccc995758177fb425f1c2d30e71adfc1d00aa5"
            ],
            "markers": "platform_system == 'Linux'",
            "version": "==3.3.1"
        }
    },
    "develop": {}
}

Sep 01 '20 20:09 jules-ch

Cross-platform dependency resolution is problematic and can fail in many cases. For example, the package has only sdist and it has some platform-specific requirements to build it. Locking will fail because the requirements can't be met on other platforms.

So the decision is made to not resolve the sub-dependencies at all. You can notice the lock file generated by 2018.11.26 is also incorrect: All sub-dependencies of python-ldap isn't marked as Linux only.

So under the current situation, you should lock dependencies on Linux.

Sep 02 '20 04:09 frostming

I would propose that there is at least some feedback shown when sub-dependencies are omitted while locking. The current behavior is unexpected from the user perspective and can only be detected by manually verifying the lockfile.

Cross-platform dependency resolution is problematic and can fail in many cases. For example, the package has only sdist and it has some platform-specific requirements to build it. Locking will fail because the requirements can't be met on other platforms.

Does this mean a dependency has to be able to be built on the current system to enumerate its dependencies? If yes, what is the motivation behind it?

Sep 11 '20 17:09 Systemcluster

I would propose that there is at least some feedback shown when sub-dependencies are omitted while locking. The current behavior is unexpected from the user perspective and can only be detected by manually verifying the lockfile.

Agreed with this; it would help figure out cross-platform build issues much sooner. In the meantime, I've started hunting platform-specific sub-dependencies and pinning them with a marker in my Pipenv file, as suggested here: https://github.com/pypa/pipenv/issues/4408#issuecomment-668324177

Does this mean a dependency has to be able to be built on the current system to enumerate its dependencies? If yes, what is the motivation behind it?

I have found that, for at least packages with native dependencies like spicy, this seems true; pipenv lock will attempt to compile the native bits, and on macOS Big Sur, I sometimes have to prefix it with SYSTEM_VERSION_COMPAT=1 to ensure older packages that can't deal with Big Sur's native toolchain major version bump can still resolve correctly.

Jul 27 '21 16:07 tinkerware

Can this be rechecked with pipenv==2022.9.2?

Aug 21 '22 11:08 matteius