Pipenv>=2020 can't install local packages depending between them

[n1ngu]

Issue description

If a local package A has a dependency on another local package version B==X that is not present in the public pypi repository, the installation will fail.

The whole 2020.* series are affected. The 2022.1.8 version is also affected.

2018.11.26 was unaffected

Observations

• The fact that A is installed locally might not be relevant but it is hard to find a package that depends on a second package in a version that is not found in pypi.
• If the B package does not exist at all, the error message is about "No matching distribution found for B" instead of "There are incompatible versions in the resolved dependencies"

Expected result

The local folder containing the B==X package satisifes the A package dependency on it.

Actual result

(Edited)

Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✘ Locking Failed! 
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1093, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 818, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches B==X (from -r /tmp/pipenv4lmztgo3requirements/pipenv-7iw66_7i-constraints.txt (line 8))
Tried: [....]
There are incompatible versions in the resolved dependencies:

Steps to replicate

A minirepo is provided to replicate the issue

https://github.com/n1ngu/pipenv-issue-4553

Clone it and try to lock the dependencies:

pipenv lock --verbose

Output:

root@2ed6ff938352:/mnt# pipenv lock --verbose
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py:218: PipDeprecationWarning: DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
 pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  deprecated(
/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py:218: PipDeprecationWarning: DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
 pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  deprecated(
Resolving dependencies...
Reporter.starting()
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('nonexistent'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('nonexistent'), None)
CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement nonexistent (from versions: none)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 341, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    raise RequirementsConflicted(criterion)
pipenv.patched.notpip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('nonexistent')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 343, in resolve
    raise ResolutionImpossible(e.criterion.information)
pipenv.patched.notpip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('nonexistent'), parent=None)]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 882, in resolve
    results = resolver.resolve(self.constraints, check_supported_wheels=False)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
    raise error from e
pipenv.patched.notpip._internal.exceptions.DistributionNotFound: No matching distribution found for nonexistent
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 766, in <module>
    main()
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 760, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 743, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 704, in resolve_packages
    results, resolver = resolve(
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 685, in resolve
    return resolve_deps(
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 1377, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 1106, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 884, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: No matching distribution found for nonexistent
✘ Locking Failed! 

---

$ pipenv --support

Pipenv version: '2022.1.8'

Pipenv location: '/usr/local/lib/python3.9/site-packages/pipenv'

Python location: '/usr/local/bin/python'

Python installations found:

• 3.9.10: /usr/local/bin/python
• 3.9.10: /usr/local/bin/python3.9
• 3.9.10: /usr/local/bin/python3
• 3.9.2: /usr/bin/python3.9
• 3.9.2: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.10',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.9.16-1-MANJARO',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Mon Dec 21 22:00:46 UTC 2020',
 'python_full_version': '3.9.10',
 'python_version': '3.9',
 'sys_platform': 'linux'}

System environment variables:

• HOSTNAME
• PYTHON_VERSION
• PWD
• PYTHON_SETUPTOOLS_VERSION
• HOME
• LANG
• GPG_KEY
• TERM
• SHLVL
• PYTHON_PIP_VERSION
• PYTHON_GET_PIP_SHA256
• PS1
• PYTHON_GET_PIP_URL
• PATH
• _
• PIP_SHIMS_BASE_MODULE
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
• LANG: C.UTF-8
• PWD: /mnt

---

Contents of Pipfile ('/mnt/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
nonexistent = {path = "./nonexistent"}
foo = {path = "./foo"}

[requires]
python_version = "3.9"

[tmewett]

I'm pretty sure I'm getting this issue on pipenv-2018.11.26 and pipenv-2021.5.29. With my setup, A and B are both local packages not on PyPI, and the B dependency of A is not version-pinned.

[n1ngu]

As of pipenv-2021.11.23 this now simply crashes in the vendored requirementslib:

git clone --depth=1 https://github.com/tryton/trytond
git clone --depth=1 https://github.com/tryton/country
pipenv install ./trytond
pipenv install --verbose ./country

Edit: but apparently it has nothing to do with local packages depending between them but with pipenv being totally uncapable of installing a local copy of the example package I picked (trytond-country), regardless of its requirements being public PyPI packages or not.

[duailibe]

A minimal reproducible example of this bug: https://github.com/duailibe/pipenv-local-deps-repro

And it worked in 2018.11.26 indeed

[matteius]

@n1ngu and @duailibe Could you recheck with pipenv==2022.1.8 and if that still doesn't work we have a branch that upgrades to pip==22.0.4 which is vendor-pip-22.0.3-followup-changes Could you try those?

[n1ngu]

Hi @matteius , I just set up this minirepo for the minimal reproduction of this issue

https://github.com/n1ngu/pipenv-issue-4553

I just tried and neither pipenv-2022.1.8 nor the development vendor-pip-22.0.3-followup-changes branch version can install it.

Notice the issue occurs in the locking / dependency resolution phase. If a project is locked with pipenv-2018, future versions can install it but not relock it.

[matteius]

@n1ngu So at first look, and some background is that the locking/resolution/installation phase has all ben converted over to use pip internals. So with pip just trying to install package foo in your example is an issue because it cannot resolve the nonexistent package from the setup.cfg. It works in pip if I manually install pip install -e nonexistent/ first and then pip install -e foo/ .... I think what is happening in newer versions of pipenv is that all requirements get passed to the resolver in one go and it is just building the local dependencies in isolation so when it gets to foo, even if nonexistent was already installed and in the lockfile, it search pypi and errors out.

[matteius]

Can the assumptions and behavior of this ticket be rechecked with pipenv==2022.4.20?

[n1ngu]

Nope. Same output for both 2022.04.20 and 2022.04.21.

pipenv.exceptions.ResolutionFailure: ERROR: No matching distribution found for nonexistent

[matteius]

This should be resolved now on pipenv==2022.8.19

[n1ngu]

Sorry but I could reproduce the same error as always with pipenv 2022.8.19.

Just clone my minirepo and do docker build . https://github.com/n1ngu/pipenv-issue-4553

pipenv.exceptions.ResolutionFailure: ERROR: No matching distribution found for nonexistent
⠼ Locking..✘ Locking Failed! 

[matteius]

@n1ngu That is odd, I agree your example is still experiencing an issue, but we have a very similar example that does work now: https://github.com/pypa/pipenv/issues/4323

I haven't seen why your example is different yet.

[matteius]

@n1ngu It works in 2022.8.19 if you mark the packages as editable in your Pipfile:

matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-issue-4553$ git diff
diff --git a/Pipfile b/Pipfile
index 1f91e94..12867d4 100644
--- a/Pipfile
+++ b/Pipfile
@@ -6,8 +6,8 @@ verify_ssl = true
 [dev-packages]
 
 [packages]
-nonexistent = {path = "./nonexistent"}
-foo = {path = "./foo"}
+nonexistent = {editable = true, path = "./nonexistent"}
+foo = {editable = true, path = "./foo"}
 
 [requires]
 python_version = "3.9"