pipenv
pipenv copied to clipboard
pypa
Editable installs that depend on each other and use flit build system do not resolve due to lack of Requirement.name
Hi everyone,
I have a Pipfile in my project root that's similar to this:
[packages]
mypackage = {path = "packages/mypackage", editable = true}
mypackage-cli = {path = "packages/mypackage-cli", editable = true}
mypackage-cli also depends on mypackage. However, if the version of mypackage isn't yet on Pypi, pipenv is unable to make the lockfile. I'm hoping it would satisfy mypackage-cli's dependency on mypackage through the mypackage editable install.
Is there a way to do this with Pipenv?
Thank you!
@gitpushdashf It would be helpful to see some example that we can triage this with. Here is an example report that is somewhat similar in nature that has an example, we could use something similar for this report: https://github.com/pypa/pipenv/issues/5118
I modified the other example I linked to so both editable packages were in their own directory and then I pointed one of them to depend on the other via install_requires= and it in fact doesn't work. I tried downgrading to an earlier release this year before index restricted package change and it has the same issue:
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5142/scrambled_eggs$ pipenv lock -v
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Reporter.starting()
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('french_toast'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('french_toast'), None)
CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement french_toast (from versions: none)
Traceback (most recent call last):
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 341, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
raise RequirementsConflicted(criterion)
pipenv.patched.notpip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('french_toast')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 472, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 343, in resolve
raise ResolutionImpossible(e.criterion.information)
pipenv.patched.notpip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('french_toast'), parent=None)]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/utils.py", line 882, in resolve
results = resolver.resolve(self.constraints, check_supported_wheels=False)
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
raise error from e
pipenv.patched.notpip._internal.exceptions.DistributionNotFound: No matching distribution found for french_toast
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/resolver.py", line 766, in <module>
main()
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/resolver.py", line 760, in main
_main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/resolver.py", line 743, in _main
resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/resolver.py", line 704, in resolve_packages
results, resolver = resolve(
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/resolver.py", line 685, in resolve
return resolve_deps(
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/utils.py", line 1377, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/utils.py", line 1106, in actually_resolve_deps
resolver.resolve()
File "/home/matteius/.local/lib/python3.9/site-packages/pipenv/utils.py", line 884, in resolve
raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: No matching distribution found for french_toast
✘ Locking Failed!
The only work around that I can think for now is to not include the install_requires in the setup.py of the other project as that is what is causing it -- having two editable dependencies that depend on each other without explicitly defining it does work, but yeah seems like there is an edge case to potentially improve with this report.
Thank you! Let me get back to you with an example repo. We're actually using flit, but definitely a similar setup.
Here it is: https://github.com/gitpushdashf/pipenv-flit-monorepo-example
Can just clone and run pipenv lock to reproduce.
@gitpushdashf Awesome--thanks so much for putting that together. Here is what I found so far--while testing the latest pipenv version and I guessed maybe there was an order of operations thing so I renamed fooy to aooy and it changed the behavior such that then I found an import error in an exception class on the main branch. So I patched that locally but was running up against something else, so I wondered if I downgraded pipenv what would my experience be like. I found that on earlier versions this year (with the aooy package renamed still) that there was a different immediate error about your example project that suggests a pyproject.toml was not sufficient on earlier versions of pip. This Stack Overflow verifies that pip 22.x is the first pip to support jusy pyproject.toml: https://stackoverflow.com/questions/69711606/how-to-install-a-package-using-pip-in-editable-mode-with-pyproject-toml
pipenv.patched.notpip._internal.exceptions.InstallationError: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /home/matteius/pipenv-triage/pipenv-5142/pipenv-flit-monorepo-example/packages/aooy
(A "pyproject.toml" file was found, but editable mode currently requires a setuptools-based build.)
So apparently when we upgraded to pip 22.0.4 in 2022.4.21 we began to allow pyproject.toml only builds as a side-effect, but I wonder if there is a bug in requirementslib around this, because in the aooy example, its getting further in locking but for the editable installs there name attribute on the Requirement is None and it causes all kinds of issues at that point. I also note the pyptoject.toml in the example repo doesn't call out what version it is.
So I added setup.py in addition to the pyproject.toml and it ran into the same issue with the lack of name attribute in the Requirement object. I decided to delete the pyproject.toml just to see (leaving only the setup.py files) and whoa, it worked!
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5142/pipenv-flit-monorepo-example$ pipenv lock
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Warning: INFO:pep517.envbuild:Temporary build environment: /tmp/pep517-build-env-g192tjas
INFO:pep517.envbuild:Temporary build environment: /tmp/pep517-build-env-i2b50zr8
Updated Pipfile.lock (27090e)!
The resulting Pipfile.lock looks like this:
{
"_meta": {
"hash": {
"sha256": "046e336530d7241a05990b14abc137ebe1ab6d08354c54c111e30bd0fad34c5f"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.10"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.org/simple",
"verify_ssl": true
}
]
},
"default": {
"aooy": {
"editable": true,
"path": "./packages/aooy"
},
"foo-cli": {
"editable": true,
"path": "./packages/foo-cli"
}
},
"develop": {}
}
Now I am intrigued and switch the name aooy back to fooy but leaving just the setup.py and do another test and it worked as well:
{
"_meta": {
"hash": {
"sha256": "20bfde650cece080759a66be4c02dcb394c0ad4c6a36362230f101bf0122ff0c"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.10"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.org/simple",
"verify_ssl": true
}
]
},
"default": {
"foo-cli": {
"editable": true,
"path": "./packages/foo-cli"
},
"fooy": {
"editable": true,
"path": "./packages/fooy"
}
},
"develop": {}
}
Turns out that it has nothing to do with the naming of the dependencies and everything to do with there is a bug with editable installs that have a pyproject.toml file, maybe only if they depend on eachother? The bug might be in requirementslib, it might be in packaging -- both things get vendored into pipenv. I am not sure yet what the fix will be, but I have triaged the issue and will mark it appropriately.
So I want to amend my prior statement, its not the presence of the pyproject.toml that breaks it, its the contents/build system that it uses. When I deleted that file and locked, the file was regenerated to use setuptools.build_meta:__legacy__
So I have to ask @gitpushdashf what is flit_core? I am beginning to suspect that has a bug that is not generating the full Metadata that pipenv needs to get the name of the dependency.
+++ b/packages/foo-cli/pyproject.toml
@@ -1,9 +1,3 @@
[build-system]
-requires = ["flit_core >=3.2,<4"]
-build-backend = "flit_core.buildapi"
-
-[project]
-name = "foo_cli"
-authors = [{name = "me", email = "me@me"}]
-dynamic = ["version", "description"]
-dependencies = ["fooy"]
+requires = ["setuptools", "wheel"]
+build-backend = "setuptools.build_meta:__legacy__"
Thank you so much for looking into this!
flit/flit_core is a setuptools alternative, generally recommended over setuptools for projects that don't have any compile steps.
https://github.com/pypa/flit
It's a little bit simpler/leaner. I'm not sure if there's a bug with flit_core or not, but it's generally worked quite well for me.
@gitpushdashf I believe this may be resolved now. I just tested it again on the latest version:
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5142/pipenv-flit-monorepo-example-master$ pipenv lock
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (22ff0c)!
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5142/pipenv-flit-monorepo-example-master$ cat Pipfile.lock
{
"_meta": {
"hash": {
"sha256": "20bfde650cece080759a66be4c02dcb394c0ad4c6a36362230f101bf0122ff0c"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.10"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.org/simple",
"verify_ssl": true
}
]
},
"default": {
"foo-cli": {
"editable": true,
"path": "./packages/foo-cli"
},
"fooy": {
"editable": true,
"path": "./packages/fooy"
}
},
"develop": {}
}
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5142/pipenv-flit-monorepo-example-master$ pipenv sync
Installing dependencies from Pipfile.lock (22ff0c)...
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:10
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
All dependencies are now up-to-date!
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5142/pipenv-flit-monorepo-example-master$ pipenv run pip freeze
# Editable install with no version control (foo_cli==0.0.0)
-e /home/matteius/pipenv-triage/pipenv-5142/pipenv-flit-monorepo-example-master/packages/foo-cli
# Editable install with no version control (fooy==0.0.0)
-e /home/matteius/pipenv-triage/pipenv-5142/pipenv-flit-monorepo-example-master/packages/fooy
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5142/pipenv-flit-monorepo-example-master$ pipenv --version
pipenv, version 2022.8.19