marker section seems to break package installation

[natemarks]

Be sure to check the existing issues (both open and closed!), and make sure you are running the latest version of Pipenv.

Check the diagnose documentation for common issues before posting! We may close your issue if it is very similar to one of them. Please be considerate, or be on your way.

Make sure to mention your debugging experience if the documented solution failed.

Issue description

Using the Pipfile for saltstack (https://github.com/saltstack/salt/blob/4cd0680b7f48790cc82a8074695ef8f04b99c74a/Pipfile), the lock file is missing almost all of the packages unless I remove the last Pipfile section. I can't seem to reproduce it on my mac, but I can reproduce it on my AWS workspaces Amazon Linux 2 host. If I remove the last section from the Pipfile.problem to make it the workaround file, then run update again, I get the expected lockfile

Pipfile.lock.problem.txt Pipfile.lock.work-around.txt Pipfile.problem.txt Pipfile.work-around.txt

---

pipenv --support

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/home/nmarks_nwnit_com/.local/lib/python2.7/site-packages/pipenv'

Python location: '/usr/bin/python'

Python installations found:

• 3.6.8: /home/nmarks_nwnit_com/.pyenv/versions/3.6.8/bin/python3
• 2.7.14: /usr/bin/python
• 2.7.15: /home/nmarks_nwnit_com/.pyenv/versions/2.7.15/bin/python
• 3.6.8: /home/nmarks_nwnit_com/.pyenv/versions/3.6.8/bin/python3.6m

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.14.88-88.76.amzn2.x86_64',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Mon Jan 7 18:43:26 UTC 2019',
 'python_full_version': '2.7.14',
 'python_version': '2.7',
 'sys_platform': 'linux2'}

System environment variables:

• MATE_DESKTOP_SESSION_ID
• XDG_CURRENT_DESKTOP
• PYTHONDONTWRITEBYTECODE
• LESSOPEN
• LOGNAME
• USER
• HOME
• XDG_VTNR
• PATH
• DISPLAY
• LANG
• TERM
• SHELL
• XAUTHORITY
• PYENV_SHELL
• SESSION_MANAGER
• SHLVL
• HISTSIZE
• WINDOWID
• GTK_OVERLAY_SCROLLING
• SESSIONTYPE
• PIPENV_PYTHON
• QT_AUTO_SCREEN_SCALE_FACTOR
• PIPENV_ROOT
• XDG_RUNTIME_DIR
• QT_SCALE_FACTOR
• SSH_AUTH_SOCK
• VTE_VERSION
• PIP_SHIMS_BASE_MODULE
• LC_ALL
• PIP_PYTHON_PATH
• GPG_TTY
• XDG_SESSION_ID
• DBUS_SESSION_BUS_ADDRESS
• _
• DESKTOP_SESSION
• PIP_DISABLE_PIP_VERSION_CHECK
• OLDPWD
• HOSTNAME
• PYENV_ROOT
• HISTCONTROL
• PWD
• PYTHONFINDER_IGNORE_UNSUPPORTED
• COLORTERM
• MAIL
• LS_COLORS
• XDG_SEAT

Pipenv–specific environment variables:

• PIPENV_PYTHON: /home/nmarks_nwnit_com/.pyenv/shims/python
• PIPENV_ROOT: /home/nmarks_nwnit_com/.local/bin

Debug–specific environment variables:

• PATH: /home/nmarks_nwnit_com/.pyenv/shims:/home/nmarks_nwnit_com/.local/bin:/home/nmarks_nwnit_com/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/sbin:/usr/local/sbin:/usr/sbin:/home/nmarks_nwnit_com/pycharm-2018.3.4/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /home/nmarks_nwnit_com/PycharmProjects/salt

---

Contents of Pipfile ('/home/nmarks_nwnit_com/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]

[requires]
python_version = "2.7"

Contents of Pipfile.lock ('/home/nmarks_nwnit_com/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "ae4bdd7d4157baab65ae9d0e8389a6011e6b640995372c45ec81fa5d1ddfae9f"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "2.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

[natemarks]

I can also reproduce this in a centos docker container. I attached the Dockerfile and the files it uses. I built the container with:

docker build --network=host -t "centos_dev_env:dockerfile"

Then I downloaded the saltstack Pipfile curl -o Pipfile https://raw.githubusercontent.com/saltstack/salt/develop/Pipfile

Then ran install and it failed to install just about everything

[root@a-26uyegg5fek5x tests_salt_pipenv]# pipenv install --dev Creating a virtualenv for this project... Pipfile: /tests_salt_pipenv/Pipfile Using /usr/bin/python (2.7.5) to create virtualenv... ⠙ Creating virtual environment...Already using interpreter /usr/bin/python New python executable in /root/.local/share/virtualenvs/tests_salt_pipenv-0hIaiOjn/bin/python Installing setuptools, pip, wheel... done.

✔ Successfully created virtual environment! Virtualenv location: /root/.local/share/virtualenvs/tests_salt_pipenv-0hIaiOjn Pipfile.lock not found, creating... Locking [dev-packages] dependencies... ✔ Success! Locking [packages] dependencies... ✔ Success! Updated Pipfile.lock (07c7e7)! Installing dependencies from Pipfile.lock (07c7e7)... DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. Ignoring httpretty: markers 'python_version >= "3.4"' don't match your environment 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 3/3 — 00:00:00 To activate this project's virtualenv, run pipenv shell. Alternatively, run a command inside the virtualenv with pipenv run. [root@a-26uyegg5fek5x tests_salt_pipenv]# pipenv run pip list DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. Package Version

---

futures 3.2.0
pip 19.0.3 setuptools 40.8.0 six 1.12.0 wheel 0.33.1

centos-pyenv-pipenv.tar.gz

[natemarks]

Oops I forgot to mention that then, when I remove the last section and run pipenv install --dev again it install s everything

[frostming]

@natemarks Oh, it's a tomlkit parsing issue. Thanks

[frostming]

@natemarks This needs some patch of the upstream project. At this point, please avoid defining sub tables after another table -- make the sub tables directly follow the parent table. Here is the fixed Pipfile which can work:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
Jinja2 = "*"
msgpack-python = ">0.3,!=0.5.5"
PyYAML = "*"
MarkupSafe = "*"
requests = ">=1.0.0"
tornado = ">=4.2.1,<5.0"
pycrypto = ">=2.6.1"
pyzmq = ">=2.2.0"

[packages.futures]
# Required by Tornado to handle threads stuff.
version = ">=2.0"
markers = "python_version < '3.0'"

[dev-packages]
mock = ">=2.0.0"
apache-libcloud = ">=0.14.0"
boto = ">=2.32.1"
boto3 = ">=1.2.1"
moto = ">=0.3.6"
SaltPyLint = ">=v2017.3.6"
pytest = ">=4.0.1"
pytest-cov = "*"
pytest-salt = "==2018.12.8"
pytest-timeout = ">=1.3.3"
pytest-tempdir = ">=2018.8.11"
pytest-helpers-namespace = ">=2017.11.11"

[dev-packages.httpretty]
# httpretty Needs to be here for now even though it's a dependency of boto.
# A pip install on a fresh system will decide to target httpretty 0.8.10 to
# satisfy other requirements, and httpretty 0.8.10 has bugs in setup.py that
# prevent it from being successfully installed (at least on Python 3.4).
version = "*"
markers = "python_version >= '3.4'"

[techalchemy]

@frostming so this needs to be handled in tomlkit? do we have an issue opened there? if not, can one of you make sure one gets filed? Thanks for the report on this one

[frostming]

@techalchemy Yes, https://github.com/sdispater/tomlkit/issues/42

[bakhtiary]

This seems to be another version of: https://github.com/pypa/pipenv/issues/4866

This was fixed in the previous release. I have tested the original saltstack file that was failing: https://github.com/saltstack/salt/blob/4cd0680b7f48790cc82a8074695ef8f04b99c74a/Pipfile

It works both with main and with version 2022.8.5. Also tested it with version 2022.5.2 and indeed it does break.

We can close this I think.