Locking fails under specific circumstances with private dev dependencies

[monkeyman192]

Issue description

I have a pipfile which is roughly like this:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
name = "private"
url = "private_url"
verify_ssl = true

[dev-packages]
private_dev_repo = {version="*", index="private"}

[packages]
private_repo = {version="*", index="private"}

[requires]
python_version = "3.9"

In this case private_dev_repo also has private_repo as a dependency. I am unable to lock this pipfile (ie. pipenv lock --dev) as it fails to resolve dependencies for dev libraries. However, if I add private_repo = {version="*", index="private"} to the dev list it will succeed. I also can do pipenv install --dev --skip-lock fine, so it's just locking that fails.

Expected result

I would expect that since the non-dev packages are locked before dev packages, any dependencies which are resolved there would also be used for the dev packages.

Actual result

Dependencies fail to be resolved.

Steps to replicate

See above description for steps. I haven't tested on older versions to see if this is a regression, however I can investigate further if required. This was tested on the latest version (2023.7.4)

[matteius]

However, if I add private_repo = {version="*", index="private"} to the dev list it will succeed.

This is expected, as you have your default index as pypi and the package groups or categories get locked independently. The dev-group is constrained by the default group but that doesn't help the pip resolver any.

[monkeyman192]

This seems... unintuitive? Is this documented somewhere? To me it seems like it would make more sense to re-use the already found dependencies in a "global" sense, otherwise I need to duplicate dependencies across both dev and non-dev packages which just looks wrong...

[matteius]

Any documentation on the topic lives here: https://pipenv.pypa.io/en/latest/indexes/

The docs probably stands to be improved, but the problem with the named package categories (in this case default and dev), is it is totally possible to specify you want to a different versions in different package groups. While default does constrain the other groups in terms of pip constraints.txt and version resolution, lets consider instead this example, say I had:

[groupA]
private_repo = {version="1.x", index="private"}

[groupB]
private_repo = {version="2.x", index="private"}

Well, my initial take is that it wouldn't be possible when locking to reference the Pipfile entries from the other group and still get good results.

There are kind of a number of factors here that would make this hard to sort out; on the surface it seems you could infer if groupB didnt have private-repo it could take the Pipfile entry from the default group, this might be reasonable seeing as default does constrain the other groups (there have been some requests to be able to override this behavior to not have default constrain other groups though).

In any case, it seems related to the other similar issues around private package indexes and the desired ability for transient dependencies to more easily specify what index they should be able to pull from.

Not to get into the weeds on this, but it kind of requires pip make some additional decisions too about how to handle better index restrictions -- there is discussion of removing the ability to scan multiple indexes (or alternative proposals) from pip in order to ensure all users are protected from dependency confusion attacks.

[kalebmckale]

@matteius On this topic, if I'm understanding correctly what you've said here: pipenv locks each category independently, i.e. there is no resolution between dependencies of categories? So, does that mean two categories can have conflicting dependencies if both installed?

[matteius]

@kalebmckale the only category the constraints the other categories currently is the default one -- some people have complained on both sides of this which makes me think that should become a pipenv configuration directive (in the Pipfile) whether or not default will constrain the other groups. Possibly there is a use case where you want constraints across categories other than default, but that gets somewhat complicated.

[kalebmckale]

Understandable. I think this is probably fine for my use case.

I tend to use categories to mirror optional dependencies in pyproject.toml.

[monkeyman192]

@kalebmckale the only category the constraints the other categories currently is the default one -- some people have complained on both sides of this which makes me think that should become a pipenv configuration directive (in the Pipfile) whether or not default will constrain the other groups. Possibly there is a use case where you want constraints across categories other than default, but that gets somewhat complicated.

The idea of having a flag which is something like "globally constrained" would be good I think? Then if people want to have everything constrained by the one set of of constraints it would be easy. I think that having any more granularity than that would be too complex like you say.

[kalebmckale]

Ideally, it would be easy. However, I suspect that it may also cause the pip Resolver to fail or give up if the number of dependencies / graph gets too large. Alas, it's out of my depth; so hopefully you'll consider submitting a PR for it.