pipenv icon indicating copy to clipboard operation
pipenv copied to clipboard

Published 20 hours ago • verified publisher icon pypa

Quoted names aren't fully normalised in Pipfile.lock

Open greysteil opened this issue 7 years ago • 9 comments

Issue description

When a dependency name is quoted in the Pipfile its entry in the Pipfile.lock is not (fully) normalised.

Expected result

Quoted names would not be treated differently to other names.

Actual result

Quoted names are downcased, but any . characters are not substituted with - characters.

Steps to replicate

Calling pipenv lock on the below Pipfile will replicate:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
"Discord.py" = "==0.16.1"

The key in the lockfile will then be discord.py rather than the expected discord-py.

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.6'

Other Python installations in PATH:

  • 2.7: /usr/bin/python2.7

  • 2.7: /usr/bin/python2.7

  • 3.6: /usr/local/bin/python3.6m

  • 3.6: /usr/local/bin/python3.6

  • 3.6: /usr/local/bin/python3.6

  • 3.6.5: /usr/local/bin/python

  • 3.6.5: /usr/local/bin/python

  • 2.7.10: /usr/bin/python

  • 3.6.5: /usr/local/bin/python3

  • 3.6.5: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT '
                     '2018; root:xnu-4570.71.2~1/RELEASE_X86_64',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

  • TERM_PROGRAM
  • PYENV_ROOT
  • TERM
  • SHELL
  • CLICOLOR
  • TMPDIR
  • Apple_PubSub_Socket_Render
  • TERM_PROGRAM_VERSION
  • TERM_SESSION_ID
  • USER
  • SSH_AUTH_SOCK
  • LSCOLORS
  • PATH
  • PWD
  • EDITOR
  • LANG
  • XPC_FLAGS
  • RBENV_SHELL
  • XPC_SERVICE_NAME
  • HOME
  • SHLVL
  • LOGNAME
  • OLDPWD
  • _
  • __CF_USER_TEXT_ENCODING
  • PYTHONDONTWRITEBYTECODE
  • PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /Users/greysteil/.yarn/bin:/Users/greysteil/.config/yarn/global/node_modules/.bin:/Users/greysteil/go/bin::/usr/local/opt/erlang@20/bin:/Users/greysteil/.pyenv/bin:/Users/greysteil/.cargo/bin:/usr/local/heroku/bin:/Users/greysteil/.rbenv/shims:/usr/local/bin:./node_modules/.bin:.bundle/binstubs:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/go/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/usr/local/git/bin:/usr/local/sbin
  • SHELL: /bin/bash
  • EDITOR: subl -w
  • LANG: en_GB.UTF-8
  • PWD: /Users/greysteil/code/python-test

Contents of Pipfile ('/Users/greysteil/code/python-test/Pipfile'):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
"Discord.py" = "==0.16.1"

Contents of Pipfile.lock ('/Users/greysteil/code/python-test/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "a007c23266ee47badb0c611c2baa6da5158a56a1fffce80eaff72101bea1d354"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "aiohttp": {
            "hashes": [
                "sha256:3a253332a0d8f82549e65035ebe7199580c3ea0e47071b7428f25b109b3c0310",
                "sha256:3ac6fa105355928e2fc02e876d0d72d6230557d4637017ebf09aec7611124155",
                "sha256:3bfcb76553d7f6296d1a598162d5fb890198f98c021540cbbb85bb604ff198db",
                "sha256:714c62532ca6be90be4b54002743e7ea277ec78b45f04ae86cdc6f45a8400abd",
                "sha256:81a6aaace2b9e8a87531277a5d5f998efbd3554c15bf47173834386966d1bbe1",
                "sha256:ad374a5d7be1de271ecd0fb0ef87c0f8dd9fb062e6fae350fa6c098360018978",
                "sha256:c3e1897726f97d40e067e8b658b2dbdfe216f32b801c5c589212e1b1f9aa8388",
                "sha256:c579ec606f25b3f756f177fee6db344f8d7ef75cfc0603a94c9fa1d1c645789d",
                "sha256:e1985766a4c83fcbdf7dde06544231fc9fb3de8929788179e623d6f9f9f321d2"
            ],
            "version": "==1.0.5"
        },
        "async-timeout": {
            "hashes": [
                "sha256:474d4bc64cee20603e225eb1ece15e248962958b45a3648a9f5cc29e827a610c",
                "sha256:b3c0ddc416736619bd4a95ca31de8da6920c3b9a140c64dbef2b2fa7bf521287"
            ],
            "markers": "python_version >= '3.5.3'",
            "version": "==3.0.0"
        },
        "chardet": {
            "hashes": [
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
            ],
            "version": "==3.0.4"
        },
        "discord.py": {
            "hashes": [
                "sha256:224c2ae10b2991cf7b5cbb77107b82b203b07a7bb43fdeab652a4ae69fecb24c"
            ],
            "version": "==0.16.1"
        },
        "multidict": {
            "hashes": [
                "sha256:112eeeddd226af681dc82b756ed34aa7b6d98f9c4a15760050298c21d715473d",
                "sha256:13b64ecb692effcabc5e29569ba9b5eb69c35112f990a16d6833ec3a9d9f8ec0",
                "sha256:1725373fb8f18c2166f8e0e5789851ccf98453c849b403945fa4ef59a16ca44e",
                "sha256:2061a50b7cae60a1f987503a995b2fc38e47027a937a355a124306ed9c629041",
                "sha256:35b062288a9a478f627c520fd27983160fc97591017d170f966805b428d17e07",
                "sha256:467b134bcc227b91b8e2ef8d2931f28b50bf7eb7a04c0403d102ded22e66dbfc",
                "sha256:475a3ece8bb450e49385414ebfae7f8fdb33f62f1ac0c12935c1cfb1b7c1076a",
                "sha256:49b885287e227a24545a1126d9ac17ae43138610713dc6219b781cc0ad5c6dfc",
                "sha256:4c95b2725592adb5c46642be2875c1234c32af841732c5504c17726b92082021",
                "sha256:4ea7ed00f4be0f7335c9a2713a65ac3d986be789ce5ebc10821da9664cbe6b85",
                "sha256:5e2d5e1d999e941b4a626aea46bdc4206877cf727107fdaa9d46a8a773a6e49b",
                "sha256:8039c520ef7bb9ec7c3db3df14c570be6362f43c200ae9854d2422d4ffe175a4",
                "sha256:81459a0ebcca09c1fcb8fe887ed13cf267d9b60fe33718fc5fd1a2a1ab49470a",
                "sha256:847c3b7b9ca3268e883685dc1347a4d09f84de7bd7597310044d847590447492",
                "sha256:8551d1db45f0ca4e8ec99130767009a29a4e0dc6558a4a6808491bcd3472d325",
                "sha256:8fa7679ffe615e0c1c7b80946ab4194669be74848719adf2d7867b5e861eb073",
                "sha256:a42a36f09f0f907579ff0fde547f2fde8a739a69efe4a2728835979d2bb5e17b",
                "sha256:a5fcad0070685c5b2d04b468bf5f4c735f5c176432f495ad055fcc4bc0a79b23",
                "sha256:ae22195b2a7494619b73c01129ddcddc0dfaa9e42727404b1d9a77253da3f420",
                "sha256:b360e82bdbbd862e1ce2a41cc3bbd0ab614350e813ca74801b34aac0f73465aa",
                "sha256:b96417899344c5e96bef757f4963a72d02e52653a4e0f99bbea3a531cedac59f",
                "sha256:b9e921140b797093edfc13ac08dc2a4fd016dd711dc42bb0e1aaf180e48425a7",
                "sha256:c5022b94fc330e6d177f3eb38097fb52c7df96ca0e04842c068cf0d9fc38b1e6",
                "sha256:cf2b117f2a8d951638efc7592fb72d3eeb2d38cc2194c26ba7f00e7190451d92",
                "sha256:d79620b542d9d0e23ae9790ca2fe44f1af40ffad9936efa37bd14954bc3e2818",
                "sha256:e2860691c11d10dac7c91bddae44f6211b3da4122d9a2ebb509c2247674d6070",
                "sha256:e3a293553715afecf7e10ea02da40593f9d7f48fe48a74fc5dd3ce08a0c46188",
                "sha256:e465be3fe7e992e5a6e16731afa6f41cb6ca53afccb4f28ea2fa6457783edf15",
                "sha256:e6d27895ef922bc859d969452f247bfbe5345d9aba69b9c8dbe1ea7704f0c5d9"
            ],
            "markers": "python_version >= '3.4.1'",
            "version": "==4.4.0"
        },
        "websockets": {
            "hashes": [
                "sha256:09dfec40e9b73e8808c39ecdbc1733e33915a2b26b90c54566afc0af546a9ec3",
                "sha256:2aa6d52264cecb08d39741e8fda49f5ac4872aef02617230c84d02e861f3cc5a",
                "sha256:2f5b7f3920f29609086fb0b63552bb1f86a04b8cbdcc0dbf3775cc90d489dfc8",
                "sha256:3d38f76f71654268e5533b45df125ff208fee242a102d4b5ca958da5cf5fb345",
                "sha256:3fcc7dfb365e81ff8206f950c86d1e73accdf3be2f9110c0cb73be32d2e7a9a5",
                "sha256:4128212ab6f91afda03a0c697add261bdf6946b47928db83f07298ea2cd8d937",
                "sha256:43e5b9f51dd0000a4c6f646e2ade0c886bd14a784ffac08b9e079bd17a63bcc5",
                "sha256:4a932c17cb11c361c286c04842dc2385cc7157019bbba8b64808acbc89a95584",
                "sha256:5ddc5fc121eb76771e990f071071d9530e27d20e8cfb804d9f5823de055837af",
                "sha256:7347af28fcc70eb45be409760c2a428f8199e7f73c04a621916c3c219ed7ad27",
                "sha256:85ae1e4b36aa2e90de56d211d2de36d7c093d00277a9afdd9b4f81e69c0214ab",
                "sha256:8a29100079f5b91a72bcd25d35a7354db985d3babae42d00b9d629f9a0aaa8ac",
                "sha256:a7e7585c8e3c0f9277ad7d6ee6ccddc69649cd216255d5e255d68f90482aeefa",
                "sha256:aa42ecef3aed807e23218c264b1e82004cdd131a6698a10b57fc3d8af8f651fc",
                "sha256:b19e7ede1ba80ee9de6f5b8ccd31beee25402e68bef7c13eeb0b8bc46bc4b7b7",
                "sha256:c4c5b5ce2d66cb0cf193c14bc9726adca095febef0f7b2c04e5e3fa3487a97a4",
                "sha256:de743ef26b002efceea7d7756e99e5d38bf5d4f27563b8d27df2a9a5cc57340a",
                "sha256:e1e568136ad5cb6768504be36d470a136b072acbf3ea882303aee6361be01941",
                "sha256:e8992f1db371f2a1c5af59e032d9dc7c1aa92f16241efcda695b7d955b4de0c2",
                "sha256:e9c1cdbb591432c59d0b5ca64fd30b6d517024767f152fc169563b26e7bcc9da"
            ],
            "version": "==3.4"
        }
    },
    "develop": {}
}

greysteil avatar Sep 10 '18 16:09 greysteil

(I'm still confused every time I see dashes instead of dots...)

techalchemy avatar Oct 07 '18 08:10 techalchemy

I just realised this quirk actually helped us dodge the dot-in-package-name bullet. It would have broken everything had we got this right… wow.

uranusjr avatar Oct 12 '18 18:10 uranusjr

Yeah, I thought that. Crazy! Feel slightly bad for suggesting it should be “fixed”.

greysteil avatar Oct 12 '18 19:10 greysteil

yep.... kinda happy about this one atm, just merged the fixes

techalchemy avatar Oct 12 '18 19:10 techalchemy

What's up with replacing the dot with a dash? Packages like RPi.GPIO have a dot in them.

AfzalivE avatar Nov 30 '18 03:11 AfzalivE

@AfzalivE https://www.python.org/dev/peps/pep-0503/#normalized-names

uranusjr avatar Nov 30 '18 04:11 uranusjr

Interesting. So I just searched through https://pypi.org/simple/

There are about 6927 packages that contain dots. I'm not so sure pipenv must be the one to enforce PEP and break compatibility with about 6% of all pypi's packages.

AfzalivE avatar Nov 30 '18 05:11 AfzalivE

The situation is… well a bit more complicated than that, let’s say. The simple API actually enforces the PEP (of course; it is defined by the PEP), and used by pip (and other tools built on it). It has been the case for a while.

What you see here is actually caused by an old bug in pip. This bug went unnoticed due to a related quirk in pip, and only hit because Pipenv assumes pip follows the PEP. The bug in pip has already been fixed, and is planned to be included in the next release in January next year.

uranusjr avatar Nov 30 '18 05:11 uranusjr

To the main point these names aren’t being quoted on write anymore which is a regression.

techalchemy avatar Nov 30 '18 14:11 techalchemy

It works fine the way it is.

matteius@matteius-VirtualBox:~/pipenv-triage/dot-in-name$ pipenv install acme.hello
Creating a virtualenv for this project...
Pipfile: /home/matteius/pipenv-triage/dot-in-name/Pipfile
Using default python from /usr/bin/python (3.10.6) to create virtualenv...
⠸ Creating virtual environment...created virtual environment CPython3.10.6.final.0-64 in 239ms
  creator CPython3Posix(dest=/home/matteius/.virtualenvs/dot-in-name-VX0cM0kX, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/matteius/.local/share/virtualenv)
    added seed packages: pip==23.1.2, setuptools==67.7.2, wheel==0.40.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/matteius/.virtualenvs/dot-in-name-VX0cM0kX
Creating a Pipfile for this project...
Installing acme.hello...
Resolving acme.hello...
Installing...
Adding acme.hello to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (1c178954fee09a8de992843cc28c3554fbec4c1d30089d258a787c1208477bad)!
Installing dependencies from Pipfile.lock (477bad)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
matteius@matteius-VirtualBox:~/pipenv-triage/dot-in-name$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
"acme.hello" = "*"

[dev-packages]

[requires]
python_version = "3.8"
matteius@matteius-VirtualBox:~/pipenv-triage/dot-in-name$ cat Pipfile.lock 
{
    "_meta": {
        "hash": {
            "sha256": "1c178954fee09a8de992843cc28c3554fbec4c1d30089d258a787c1208477bad"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "acme.hello": {
            "hashes": [
                "sha256:5dc7d480cea31817e838565b66a845a361247a91d1603da7b31f7e73dd157af6"
            ],
            "index": "pypi",
            "version": "==0.1"
        }
    },
    "develop": {}
}
matteius@matteius-VirtualBox:~/pipenv-triage/dot-in-name$ pipenv sync 
Installing dependencies from Pipfile.lock (477bad)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
All dependencies are now up-to-date!

matteius avatar May 19 '23 23:05 matteius