pipenv Installing git dependencies over ssh often leads to corrupted pipfile.lock or pipfile dependencies

Installing git dependencies over ssh often leads to corrupted pipfile.lock or pipfile dependencies

Open Kroppeb opened this issue 5 months ago • 10 comments

Issue description

Installing git dependencies over ssh often results in errors.

Steps to replicate

Create a new project in a new folder, and run pipenv install git+ssh://[email protected]/pypa/pipenv.git#egg=pipenv

This fails. In the created lockfile contains this:

        "pipenv": {
            "git": "git+ssh://git",
            "markers": "python_version >= '3.8'",
            "ref": "d4483dd2a3cd55c45de2e3d5ca98094c67b64af3"
        },

Which results in pip trying to run the invalid command git clone --filter=blob:none --quiet 'ssh://git@d4483dd2a3cd55c45de2e3d5ca98094c67b64af3' ...

Running pipenv install git+ssh://[email protected]/pypa/pipenv.git@main#egg=pipenv instead does seem to fix the issue

$ pipenv --support

Pipenv version: '2023.12.0'

Pipenv location: 'C:\\Users\\Robbe\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\pipenv'

Python location: 'C:\\Users\\Robbe\\AppData\\Local\\Programs\\Python\\Python312\\python.exe'

OS Name: 'nt'

User pip version: '23.3.2'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.12.1',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '11',
 'platform_system': 'Windows',
 'platform_version': '10.0.22631',
 'python_full_version': '3.12.1',
 'python_version': '3.12',
 'sys_platform': 'win32'}

System environment variables:

ALLUSERSPROFILE
APPDATA
COMMONPROGRAMFILES
COMMONPROGRAMFILES(X86)
COMMONPROGRAMW6432
COMPUTERNAME
COMSPEC
DRIVERDATA
HOMEDRIVE
HOMEPATH
LOCALAPPDATA
LOGONSERVER
NUMBER_OF_PROCESSORS
ONEDRIVE
ONEDRIVECONSUMER
OS
PATH
PATHEXT
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
PROCESSOR_LEVEL
PROCESSOR_REVISION
PROGRAMDATA
PROGRAMFILES
PROGRAMFILES(X86)
PROGRAMW6432
PSMODULEPATH
PUBLIC
SESSIONNAME
SYSTEMDRIVE
SYSTEMROOT
TEMP
TMP
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERNAME
USERPROFILE
WINDIR
WSLENV
WT_PROFILE_ID
WT_SESSION
ZES_ENABLE_SYSMAN
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv?specific environment variables:

Debug?specific environment variables:

PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\Git\cmd;C:\Program Files (x86)\Gpg4win\..\GnuPG\bin;C:\Program Files\nodejs\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Users\Robbe\AppData\Local\Programs\Python\Python312\Scripts\;C:\Users\Robbe\AppData\Local\Programs\Python\Python312\;C:\Users\Robbe\AppData\Local\Programs\Python\Python311\Scripts\;C:\Users\Robbe\AppData\Local\Programs\Python\Python311\;C:\Users\Robbe\AppData\Local\Programs\Python\Python310\Scripts\;C:\Users\Robbe\AppData\Local\Programs\Python\Python310\;C:\Users\Robbe\AppData\Local\Microsoft\WindowsApps;C:\Users\Robbe\AppData\Local\JetBrains\Toolbox\scripts;C:\Users\Robbe\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Robbe\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin;C:\Users\Robbe\AppData\Roaming\npm;C:\Users\Robbe\AppData\Local\Microsoft\WindowsApps;

Contents of Pipfile ('C:\Users\Robbe\Personal\pipenv-bug-test1\Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.12"

Contents of Pipfile.lock ('C:\Users\Robbe\Personal\pipenv-bug-test1\Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "86fe84858bd554474429681a04a3dca37e6755bcbfdc43990e5217ea4cfa89b3"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.12"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:0569859f95fc761b18b45ef421b1290a0f65f147e92a1e5eb3e635f9a5e4e66f",
                "sha256:dc383c07b76109f368f6106eee2b593b04a011ea4d55f652c6ca24a754d1cdd1"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==2024.2.2"
        },
        "distlib": {
            "hashes": [
                "sha256:034db59a0b96f8ca18035f36290806a9a6e6bd9d1ff91e45a7f172eb17e51784",
                "sha256:1530ea13e350031b6312d8580ddb6b27a104275a31106523b8f123787f494f64"
            ],
            "version": "==0.3.8"
        },
        "filelock": {
            "hashes": [
                "sha256:521f5f56c50f8426f5e03ad3b281b490a87ef15bc6c526f168290f0c7148d44e",
                "sha256:57dbda9b35157b05fb3e58ee91448612eb674172fab98ee235ccb0b5bee19a1c"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==3.13.1"
        },
        "pipenv": {
            "git": "git+ssh://git",
            "markers": "python_version >= '3.8'",
            "ref": "d4483dd2a3cd55c45de2e3d5ca98094c67b64af3"
        },
        "platformdirs": {
            "hashes": [
                "sha256:0614df2a2f37e1a662acbd8e2b25b92ccf8632929bc6d43467e17fe89c75e068",
                "sha256:ef0cc731df711022c174543cb70a9b5bd22e5a9337c8624ef2c2ceb8ddad8768"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==4.2.0"
        },
        "setuptools": {
            "hashes": [
                "sha256:385eb4edd9c9d5c17540511303e39a147ce2fc04bc55289c322b9e5904fe2c05",
                "sha256:be1af57fc409f93647f2e8e4573a142ed38724b8cdd389706a867bb4efcf1e78"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==69.0.3"
        },
        "virtualenv": {
            "hashes": [
                "sha256:4238949c5ffe6876362d9c0180fc6c3a824a7b12b80604eeb8085f2ed7460de3",
                "sha256:bf51c0d9c7dd63ea8e44086fa1e4fb1093a31e963b86959257378aef020e1f1b"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==20.25.0"
        }
    },
    "develop": {}
}

Feb 02 '24 12:02 Kroppeb

So the weird thing is that when we install our own private packages like this, this usually doesn't fail. It's often later when installing the project on a new system that issues arise. Not sure why installing pipenv seems to fail more reliably.

Feb 02 '24 12:02 Kroppeb

The problem is installing pipenv with pipenv -- it is not supported and it won't work right.

Feb 02 '24 15:02 matteius

I don't see why installing pipenv with pipenv is an issue? It's not that I'm using the virtualenv created by pipenv.

I have also been able to reproduce it with pipenv install git+ssh://[email protected]/cglwn/leftpad-pypi.git#egg=leftpad (and again, I can't using pipenv install git+ssh://[email protected]/cglwn/leftpad-pypi.git@master#egg=leftpad

Feb 02 '24 20:02 Kroppeb

Sounds like your issue relates more to ssh cloning, because you need an ssh key that gives you access to that project. Typically you want to use https urls for cloning since they are publicly available; for example pipenv hasn't granted you explicit access, to the repository so ssh is going to fail.

Feb 02 '24 20:02 matteius

No, not the issue. As mentioned in my report in the pipenv.lock file the following appears: "git": "git+ssh://git" instead of the expected "git" = "git+ssh://[email protected]/cglwn/leftpad-pypi.git" in case of leftpad.

As a result when trying to make pip install it, the following requirement is created: leftpad@ git+ssh://git@7184e2cd37256058cbece2aeebf4bff2533bd699 instead of leftpad@ git+ssh://[email protected]/cglwn/leftpad-pypi.git@7184e2cd37256058cbece2aeebf4bff2533bd699

Additionally, if you fix the pipenv.lock file manually, everything seems to work. But the requirement isn't mentioned in the pipenv file. Running the install command again, will again fail, but this time will add the following requirement

leftpad = {ref = "github.com/cglwn/leftpad-pypi.git", git = "git+ssh://git"}

It seems that somewhere along the way, pipenv splits the requirement on @ symbols and misinterprets the case where there is only one @ symbol.

What really bugs me is that in my experience, installing git dependencies over ssh usually works, and most issue we've encountered at my job is when cloning a project and trying to create the environment. I am really surprised that now I'm trying to trigger an issue, that it is consistently failing.

Feb 02 '24 20:02 Kroppeb

Even on pip, when I try to clone a repo with ssh that my key doesn't have access to, I get rejected.

Feb 02 '24 20:02 matteius

I have access to the repo

Feb 02 '24 20:02 Kroppeb

This worked for me, I understand its https, but I don't have access to that repo to test ssh with: $ pipenv install git+https://github.com/cglwn/leftpad-pypi.git@master

I also tested it with pydantic:

$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pydantic = {ref = "main", git = "git+https://github.com/pydantic/pydantic.git"}
leftpad = {ref = "master", git = "git+https://github.com/cglwn/leftpad-pypi.git"}

[dev-packages]

[requires]
python_version = "3.11"

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-6076
$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "1457336e0469ee62ed6e522c6c5018594aedc96868d5248a128929262bdc3b4e"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.11"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "annotated-types": {
            "hashes": [
                "sha256:0641064de18ba7a25dee8f96403ebc39113d0cb953a01429249d5c7564666a43",
                "sha256:563339e807e53ffd9c267e99fc6d9ea23eb8443c08f112651963e24e22f84a5d"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==0.6.0"
        },
        "leftpad": {
            "git": "git+https://github.com/cglwn/leftpad-pypi.git",
            "ref": "7184e2cd37256058cbece2aeebf4bff2533bd699"
        },
        "pydantic": {
            "git": "git+https://github.com/pydantic/pydantic.git",
            "markers": "python_version >= '3.8'",
            "ref": "8364920acb272e2df5c6e261ca67622f3d6d1874"
        },
        "pydantic-core": {
            "hashes": [
                "sha256:06f0d5a1d9e1b7932477c172cc720b3b23c18762ed7a8efa8398298a59d177c7",
                "sha256:07982b82d121ed3fc1c51faf6e8f57ff09b1325d2efccaa257dd8c0dd937acca",
                "sha256:0f478ec204772a5c8218e30eb813ca43e34005dff2eafa03931b3d8caef87d51",
                "sha256:102569d371fadc40d8f8598a59379c37ec60164315884467052830b28cc4e9da",
                "sha256:10dca874e35bb60ce4f9f6665bfbfad050dd7573596608aeb9e098621ac331dc",
                "sha256:150ba5c86f502c040b822777e2e519b5625b47813bd05f9273a8ed169c97d9ae",
                "sha256:1661c668c1bb67b7cec96914329d9ab66755911d093bb9063c4c8914188af6d4",
                "sha256:1a2fe7b00a49b51047334d84aafd7e39f80b7675cad0083678c58983662da89b",
                "sha256:1ae8048cba95f382dba56766525abca438328455e35c283bb202964f41a780b0",
                "sha256:20f724a023042588d0f4396bbbcf4cffd0ddd0ad3ed4f0d8e6d4ac4264bae81e",
                "sha256:2133b0e412a47868a358713287ff9f9a328879da547dc88be67481cdac529118",
                "sha256:21e3298486c4ea4e4d5cc6fb69e06fb02a4e22089304308817035ac006a7f506",
                "sha256:21ebaa4bf6386a3b22eec518da7d679c8363fb7fb70cf6972161e5542f470798",
                "sha256:23632132f1fd608034f1a56cc3e484be00854db845b3a4a508834be5a6435a6f",
                "sha256:2d5bea8012df5bb6dda1e67d0563ac50b7f64a5d5858348b5c8cb5043811c19d",
                "sha256:300616102fb71241ff477a2cbbc847321dbec49428434a2f17f37528721c4948",
                "sha256:30a8259569fbeec49cfac7fda3ec8123486ef1b729225222f0d41d5f840b476f",
                "sha256:399166f24c33a0c5759ecc4801f040dbc87d412c1a6d6292b2349b4c505effc9",
                "sha256:3fac641bbfa43d5a1bed99d28aa1fded1984d31c670a95aac1bf1d36ac6ce137",
                "sha256:42c29d54ed4501a30cd71015bf982fa95e4a60117b44e1a200290ce687d3e640",
                "sha256:462d599299c5971f03c676e2b63aa80fec5ebc572d89ce766cd11ca8bcb56f3f",
                "sha256:4eebbd049008eb800f519578e944b8dc8e0f7d59a5abb5924cc2d4ed3a1834ff",
                "sha256:502c062a18d84452858f8aea1e520e12a4d5228fc3621ea5061409d666ea1706",
                "sha256:5317c04349472e683803da262c781c42c5628a9be73f4750ac7d13040efb5d2d",
                "sha256:5511f962dd1b9b553e9534c3b9c6a4b0c9ded3d8c2be96e61d56f933feef9e1f",
                "sha256:561be4e3e952c2f9056fba5267b99be4ec2afadc27261505d4992c50b33c513c",
                "sha256:601d3e42452cd4f2891c13fa8c70366d71851c1593ed42f57bf37f40f7dca3c8",
                "sha256:644904600c15816a1f9a1bafa6aab0d21db2788abcdf4e2a77951280473f33e1",
                "sha256:653a5dfd00f601a0ed6654a8b877b18d65ac32c9d9997456e0ab240807be6cf7",
                "sha256:694a5e9f1f2c124a17ff2d0be613fd53ba0c26de588eb4bdab8bca855e550d95",
                "sha256:71b4a48a7427f14679f0015b13c712863d28bb1ab700bd11776a5368135c7d60",
                "sha256:72bf9308a82b75039b8c8edd2be2924c352eda5da14a920551a8b65d5ee89253",
                "sha256:735dceec50fa907a3c314b84ed609dec54b76a814aa14eb90da31d1d36873a5e",
                "sha256:73802194f10c394c2bedce7a135ba1d8ba6cff23adf4217612bfc5cf060de34c",
                "sha256:780daad9e35b18d10d7219d24bfb30148ca2afc309928e1d4d53de86822593dc",
                "sha256:8655f55fe68c4685673265a650ef71beb2d31871c049c8b80262026f23605ee3",
                "sha256:877045a7969ace04d59516d5d6a7dee13106822f99a5d8df5e6822941f7bedc8",
                "sha256:87bce04f09f0552b66fca0c4e10da78d17cb0e71c205864bab4e9595122cb9d9",
                "sha256:8d4dfc66abea3ec6d9f83e837a8f8a7d9d3a76d25c9911735c76d6745950e62c",
                "sha256:8ec364e280db4235389b5e1e6ee924723c693cbc98e9d28dc1767041ff9bc388",
                "sha256:8fa00fa24ffd8c31fac081bf7be7eb495be6d248db127f8776575a746fa55c95",
                "sha256:920c4897e55e2881db6a6da151198e5001552c3777cd42b8a4c2f72eedc2ee91",
                "sha256:920f4633bee43d7a2818e1a1a788906df5a17b7ab6fe411220ed92b42940f818",
                "sha256:9795f56aa6b2296f05ac79d8a424e94056730c0b860a62b0fdcfe6340b658cc8",
                "sha256:98f0edee7ee9cc7f9221af2e1b95bd02810e1c7a6d115cfd82698803d385b28f",
                "sha256:99c095457eea8550c9fa9a7a992e842aeae1429dab6b6b378710f62bfb70b394",
                "sha256:99d3a433ef5dc3021c9534a58a3686c88363c591974c16c54a01af7efd741f13",
                "sha256:99f9a50b56713a598d33bc23a9912224fc5d7f9f292444e6664236ae471ddf17",
                "sha256:9c46e556ee266ed3fb7b7a882b53df3c76b45e872fdab8d9cf49ae5e91147fd7",
                "sha256:9f5d37ff01edcbace53a402e80793640c25798fb7208f105d87a25e6fcc9ea06",
                "sha256:a0b4cfe408cd84c53bab7d83e4209458de676a6ec5e9c623ae914ce1cb79b96f",
                "sha256:a497be217818c318d93f07e14502ef93d44e6a20c72b04c530611e45e54c2196",
                "sha256:ac89ccc39cd1d556cc72d6752f252dc869dde41c7c936e86beac5eb555041b66",
                "sha256:adf28099d061a25fbcc6531febb7a091e027605385de9fe14dd6a97319d614cf",
                "sha256:afa01d25769af33a8dac0d905d5c7bb2d73c7c3d5161b2dd6f8b5b5eea6a3c4c",
                "sha256:b1fc07896fc1851558f532dffc8987e526b682ec73140886c831d773cef44b76",
                "sha256:b49c604ace7a7aa8af31196abbf8f2193be605db6739ed905ecaf62af31ccae0",
                "sha256:b9f3e0bffad6e238f7acc20c393c1ed8fab4371e3b3bc311020dfa6020d99212",
                "sha256:ba07646f35e4e49376c9831130039d1b478fbfa1215ae62ad62d2ee63cf9c18f",
                "sha256:bd88f40f2294440d3f3c6308e50d96a0d3d0973d6f1a5732875d10f569acef49",
                "sha256:c0be58529d43d38ae849a91932391eb93275a06b93b79a8ab828b012e916a206",
                "sha256:c45f62e4107ebd05166717ac58f6feb44471ed450d07fecd90e5f69d9bf03c48",
                "sha256:c56da23034fe66221f2208c813d8aa509eea34d97328ce2add56e219c3a9f41c",
                "sha256:c94b5537bf6ce66e4d7830c6993152940a188600f6ae044435287753044a8fe2",
                "sha256:cebf8d56fee3b08ad40d332a807ecccd4153d3f1ba8231e111d9759f02edfd05",
                "sha256:d0bf6f93a55d3fa7a079d811b29100b019784e2ee6bc06b0bb839538272a5610",
                "sha256:d195add190abccefc70ad0f9a0141ad7da53e16183048380e688b466702195dd",
                "sha256:d25ef0c33f22649b7a088035fd65ac1ce6464fa2876578df1adad9472f918a76",
                "sha256:d6cbdf12ef967a6aa401cf5cdf47850559e59eedad10e781471c960583f25aa1",
                "sha256:d8c032ccee90b37b44e05948b449a2d6baed7e614df3d3f47fe432c952c21b60",
                "sha256:daff04257b49ab7f4b3f73f98283d3dbb1a65bf3500d55c7beac3c66c310fe34",
                "sha256:e83ebbf020be727d6e0991c1b192a5c2e7113eb66e3def0cd0c62f9f266247e4",
                "sha256:ed3025a8a7e5a59817b7494686d449ebfbe301f3e757b852c8d0d1961d6be864",
                "sha256:f1936ef138bed2165dd8573aa65e3095ef7c2b6247faccd0e15186aabdda7f66",
                "sha256:f5247a3d74355f8b1d780d0f3b32a23dd9f6d3ff43ef2037c6dcd249f35ecf4c",
                "sha256:fa496cd45cda0165d597e9d6f01e36c33c9508f75cf03c0a650018c5048f578e",
                "sha256:fb4363e6c9fc87365c2bc777a1f585a22f2f56642501885ffc7942138499bf54",
                "sha256:fb4370b15111905bf8b5ba2129b926af9470f014cb0493a67d23e9d7a48348e8",
                "sha256:fbec2af0ebafa57eb82c18c304b37c86a8abddf7022955d1742b3d5471a6339e"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==2.16.1"
        },
        "typing-extensions": {
            "hashes": [
                "sha256:23478f88c37f27d76ac8aee6c905017a143b0b1b886c3c9f66bc2fd94f9f5783",
                "sha256:af72aea155e91adfc61c3ae9e0e342dbc0cba726d6cba4b6c72c1f34e47291cd"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==4.9.0"
        }
    },
    "develop": {}
}

Feb 02 '24 20:02 matteius

Yeah, I wouldn't be surprised if this is a ssh only issue. I'll update the issue

Feb 02 '24 21:02 Kroppeb

I think I found the issue is colon vs slash -- not sure if github changed how they are exporting the ssh URL in the UI or not, but this has parity with how pip requires the ssh line be supplied: https://github.com/pypa/pipenv/issues/6085#issuecomment-1926901571

Feb 05 '24 12:02 matteius

pipenv pipenv copied to clipboard

Installing git dependencies over ssh often leads to corrupted pipfile.lock or pipfile dependencies

Issue description

Steps to replicate

pipenv
pipenv copied to clipboard