pipenv lock does not capture dev and non-dev requirements of the same package

[GPHemsley]

Issue description

I have a Pipfile that specifies a stable version of a package for production and an editable local version of a package for development. These differences are not being captured when running pipenv lock.

Expected result

Pipfile.lock is updated with one package definition in the "default" section and another in the "develop" section.

Actual result

The definition in the "default" section is also listed in the "develop" section.

Manually editing the Pipfile.lock allows pipenv install --dev to work as expected.

Steps to replicate

1. Update the Pipfile to have the same package in both the "packages" and "dev-packages" sections, with different definitions/requirements.
2. Run pipenv lock.
3. Observe that Pipfile.lock has not captured the "dev-packages" definition.

[GPHemsley]

Note: pipenv install --dev only works after pipenv uninstall --all. Running pipenv install --dev after having run pipenv install does not install the dev version on top of the default version. (However, the opposite does happen: running pipenv install after pipenv install --dev will replace the dev version.)

[shiyuangu]

Due to the same issue, "pipenv lock -d -r > dev-requirements.txt" may not generate a valid requirement file for pip. To reproduce the error:

1. Create a Pipfile as follows:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
"alembic" = "==1.7.5"

[dev-packages]
"flake8" = "4.0.1"

[requires]
python_version = "3.6"

2. pipenv lock -d -r > dev-requirements.txt
3. pip install -r dev-requirements.txt, this will encounter an error due to the conflicting version:

ERROR: Cannot install -r dev-requirements.txt (line 12), -r dev-requirements.txt (line 13) and importlib-metadata==4.8.2 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested importlib-metadata==4.8.2
    alembic 1.7.5 depends on importlib-metadata; python_version < "3.9"
    flake8 4.0.1 depends on importlib-metadata<4.3; python_version < "3.8"

The cause of error seems to be an intentional overwrite of dev-dependencies with default dependencies in the do_lock function

[sweco]

Assuming that it is possible to resolve dependencies and dev dependencies together without a conflict (like in the @shiyuangu's example where the solution is to use for example importlib-metadata==4.2.0), I see ~three~ four strategies how one can approach such an issue.

When I hit this issue, I used the first approach which I achieved by pinning the conflicting package in default packages. However, it would probably be nice to add a CLI option to be able to allow downgrading a default package so that the dev packages are not in conflict.

1. Downgrade the default requirement to meet the dev requirements

This would mean downgrading importlib-metadata==4.8.2 to version 4.2.0 to also meet the dev requirements (importlib-metadata<4.3). However, the main question is whether you should downgrade a default (production) package just because you want to use some linter or testing library.

2. Use different versions of the requirement in production and in development.

This can probably simply be achieved by not overwriting the dev dependencies in the do_lock function and setting up your dev environment by first installing the default packages and then the dev packages which will overwrite the versions.

3. Fail resolving the dev dependencies

When the packages resolved in Pipfile.lock cannot be installed via pipenv lock -r -d > requirements.txt && pip install -r requirements.txt, pipenv sync -d still manages to install them which can result in undefined behavior. Maybe pipenv lock should disallow locking in such cases? Or at least

pip-tools solves layered requirements as follows. They recommend to first resolve the production requirements and then use the created requirements.txt file as constraints while resolving the dev dependencies. They then fail with an error due to the unmet constraints.

1. Create requirements.in file

alembic==1.7.5

3. Run pip-compile that generates a requirements.txt file

alembic==1.7.5
    # via -r requirements.in
# ...
importlib-metadata==4.8.3
    # via
    #   alembic
    #   sqlalchemy
# ...

4. Use the requirements.txt file as constraints and observe the error

> pip install flake8==4.0.1 -c requirements.txt

ERROR: Cannot install flake8==4.0.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    flake8 4.0.1 depends on importlib-metadata<4.3; python_version < "3.8"
    The user requested (constraint) importlib-metadata==4.8.3

4. Manually pin requirement in default

This is my current solution. Upon realizing that the dev dependencies contain a conflict, I manually pin the version in the default packages.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
"alembic" = "==1.7.5"
"importlib-metadata" = "<4.3"  # Added this line to resolve the conflict.

[dev-packages]
"flake8" = "4.0.1"

[requires]
python_version = "3.6"

[matteius]

I believe this may be resolved by work @dqkqd did recently on constraint files -- could you recheck with pipenv=2022.8.19?

[sweco]

Hey @matteius, thank you for the update and sorry that it took me so long to test this.

I can definitely confirm that the behavior is different now. The generated list of requirements is now installable. However, after rerunning @shiyuangu's example, I can see that the dev requirement "flake8" = "4.0.1" is not respected. This is what the resulting dev-requirements.txt file looks like (notice flake8==5.0.4).

-i https://pypi.org/simple
alembic==1.7.5
greenlet==1.1.3
importlib-metadata==4.8.3
importlib-resources==5.4.0
mako==1.1.6
markupsafe==2.0.1
sqlalchemy==1.4.41
typing-extensions==4.1.1
zipp==3.6.0
flake8==5.0.4
importlib-metadata==4.8.3
mccabe==0.7.0
pycodestyle==2.9.1
pyflakes==2.5.0
typing-extensions==4.1.1
zipp==3.6.0

Even though the dev requirement "flake8" = "4.0.1" is not respected, I'm not sure what the correct behavior should be, because the way I see it, there are only two options, both having their pluses and minuses.

1. Respect flake8==4.0.1 which transitively requires importlib-metadata<4.3. By honoring the default requirements ("alembic==1.7.5" -> importlib-metadata) as well as dev requirements (flake8==4.0.1 -> importlib-metadata<4.3), we'd end up with importlib-metadata==4.2.0.

   ➕ All the requirements are honored, the behavior is predictable. ➖ We downgrade a default (non-dev) requirement just to respect some dev package that does not get used in production. By using an older version there is a higher chance of introducing bugs and/or security issues to your app.

2. Do not respect dev requirements. We end up with importlib-metadata==4.8.3 which is incompatible with flake8==4.0.1 but if we upgrade flake8 to 5.0.4, everything works fine.

   ➕ We don't downgrade a default (non-dev) package just because of some dev package. ➖ The behavior is less predictable since we ask for a version of a package but a different version gets selected. It would probably make sense to at least warn the user on pipenv lock -d that a dev package version was not respected.

You seem to have chosen the second approach. I cannot say which approach is clearly better. I'm maybe slightly in favor of the first approach but maybe the better solution also depends on the exact case you're solving.

What do you think about all this? Which approach do you find better? Since the second approach is now implemented , do you think a warning could be added if a requirement is not respected?