python
Installing 8.5.0 with `--no-binary=:all:` results in `LookupError`
This is a very similar issue to https://github.com/python/importlib_metadata/issues/392, but for me it is happening on py3.8.
Steps to reproduce
(on py3.8) pip install --no-binary=:all: importlib-metadata
Relevant output log
(venv) dplascen@templates$: pip install --no-binary=:all: importlib-metadata
Collecting importlib-metadata
Using cached importlib_metadata-8.5.0.tar.gz (55 kB)
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> [83 lines of output]
Collecting setuptools>=61.2
Using cached setuptools-75.3.2-py3-none-any.whl
Collecting setuptools_scm>=3.4.1 (from setuptools_scm[toml]>=3.4.1)
Using cached setuptools_scm-8.3.1.tar.gz (78 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting packaging>=20 (from setuptools_scm>=3.4.1->setuptools_scm[toml]>=3.4.1)
Using cached packaging-25.0.tar.gz (165 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting typing-extensions (from setuptools_scm>=3.4.1->setuptools_scm[toml]>=3.4.1)
Using cached typing_extensions-4.13.2.tar.gz (106 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting importlib-metadata>=4.6 (from setuptools_scm>=3.4.1->setuptools_scm[toml]>=3.4.1)
Using cached importlib_metadata-8.5.0.tar.gz (55 kB)
ERROR: Exception:
Traceback (most recent call last):
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 106, in _run_wrapper
status = _inner_run()
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 97, in _inner_run
return self.run(options, args)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
return func(self, options, args)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 386, in run
requirement_set = resolver.resolve(
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
result = self._result = resolver.resolve(
File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
failure_causes = self._attempt_to_pin_criterion(name)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
criteria = self._get_updated_criteria(candidate)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 230, in _get_updated_criteria
self._add_to_criteria(criteria, requirement, parent=candidate)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
if not criterion.candidates:
File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
return bool(self._sequence)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 174, in __bool__
return any(self)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 162, in <genexpr>
return (c for c in iterator if id(c) not in self._incompatible_ids)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 53, in _iter_built
candidate = func()
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 187, in _make_candidate_from_link
base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 233, in _make_base_candidate_from_link
self._link_candidate_cache[link] = LinkCandidate(
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in __init__
super().__init__(
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 159, in __init__
self.dist = self._prepare()
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 236, in _prepare
dist = self._prepare_distribution()
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 315, in _prepare_distribution
return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 527, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 642, in _prepare_linked_requirement
dist = _get_prepared_distribution(
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 71, in _get_prepared_distribution
with build_tracker.track(req, tracker_id):
File "/opt/homebrew/Cellar/[email protected]/3.8.19/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/operations/build/build_tracker.py", line 136, in track
self.add(req, tracker_id)
File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/operations/build/build_tracker.py", line 102, in add
raise LookupError(message)
LookupError: https://files.pythonhosted.org/packages/cd/12/33e59336dca5be0c398a7482335911a33aa0e20776128f038019f1a95f1b/importlib_metadata-8.5.0.tar.gz (from https://pypi.org/simple/importlib-metadata/) (requires-python:>=3.8) is already being built: importlib-metadata from https://files.pythonhosted.org/packages/cd/12/33e59336dca5be0c398a7482335911a33aa0e20776128f038019f1a95f1b/importlib_metadata-8.5.0.tar.gz
[end of output]
Additional context
https://github.com/pypa/setuptools-scm/issues/1131 was recently reported and closed, and may be related.
It's related
I solved the loop for setuptools scm itself
But now the build env of importlib metadata requires importlib metadata
@jaraco is there a reasonable way to provide altered pyproject.toml inside a sdist
also i strongly recommend never to use source only installs unless you have full control of the build process
and arguably anyone hitting this issue outside of distro packaging is in fact not in control of the build process
This problem is another manifestation of https://github.com/pypa/packaging-problems/issues/342. Essentially, it's not possible for build backends (or their plugins) to have dependencies that use said build backend.
I've proposed several solutions, but none of which are without tradeoffs, and many of which push the bootstrapping problem onto those environments that wish to build everything from source.
I have one approach that I'm currently exploring in the coherent.build system that addresses the problem by producing sdists that build on a more primitive build backend (like flit). If I were to migrate importlib_metadata to coherent.build (with the experimental feature enabled) (or move to flit), the problem would go away, but for importlib_metadata only. I'd still have to apply the same treatment for every dependency of setuptools_scm (and Setuptools itself once I can get it to unvendor its dependencies).
Unfortunately, support for Python 3.8 is dropped, so there's little hope of these innovations addressing the issue on Python 3.8.
There are many hacks that system integrators have used to work around the cyclic dependency issues, namely:
- avoiding use of setuptools_scm by hand-coding the version with SETUPTOOLS_SCM_PRETEND_VERSION (or similar)
- converting projects to flit before building
Another workaround that's been proposed - while bootstrapping the build backends (Setuptools+setuptools-scm in this case), ensure all of the (transitive) dependencies are present in the PYTHONPATH, either by pre-loading them from pre-built trusted installs or by manually assembling the Python modules from sources into an importable form, and then building the backend without build isolation.
It's a difficult problem to comprehend and an even more difficult problem to solve, which is why it's still broken. Good luck!
@abravalheri @jaraco can we expand setuptools sdist building in a way that allows somthing like setuptools_scm to reasonably replace build dependencies and dynamic fields
the idea would be to move the setuptools_scm dependency into something setuptools can omit for a sdist
same goes for file finer output
that way new sdists could ship whats needed in a more safe manner
I think the short answer is no, mainly because Setuptools (and by extension every backend that has dependencies that lead to itself) would need to re-write the user's sources to produce the sdist. That is, if the user has declared build-requires=['setuptools-scm'], they've declared that for the build. PEP 518 and 517 doesn't provide enough granularity for the user to indicate build-wheel-requires=['setuptools-scm'] but build-sdist-requires[]. Therefore, to prevent an installer from rightfully attempting to install a cyclic dependency, the build-requires would have to be different for sources than for source distributions. Moreover, that's just a half-step toward what coherent.build is doing in compiling the sources to a different backend for sdists. It's true that Setuptools doesn't rely on setuptools-scm except at the build from source stage (it's no longer needed at the build from sdist stage), but that guarantee doesn't hold in general; it's a special case for setuptools-scm. If Setuptools is going to special-case something for setuptools-scm, it should be to integrate it. Otherwise, whatever solution is devised should apply to any dependency. The landscape is already complicated enough without carving out special cases for specific dependencies.
As build backends can report additional dependencies the idea is more around being able to skip some if the metadata is provided by the sdist instead of rewriting the code
As build backends can report additional dependencies the idea is more around being able to skip some if the metadata is provided by the sdist instead of rewriting the code
Reading between the lines, I think you're suggesting:
- Setuptools-scm users are directed to stop specifying setuptools-scm in
build-requiresin pyproject.toml. - Setuptools adds a feature flag to enable setuptools-scm.
- When a user specifies to enable setuptools-scm, setuptools will include 'setuptools-scm' in the response to
get_build_requires_for_sdistbut notget_build_requires_for_wheel.
That sounds like a lot of work and a very special case. Currently, setuptools relies on the presence or absence of the plugin to determine whether the feature is enabled or not. If setuptools-scm were to require a different paradigm for inclusion/enablement, it also has implications for the designs of other plugins.
In any case, this issue is not a great place to be having this discussion, since the issue relates to setuptools and setuptools-scm and the Python packaging ecosystem as a whole and has nothing to do with importlib_metadata other than that importlib_metadata happens to be caught in the problem because it happens to be a dependency of a build backend's plugin.
The only reason I'm not closing this issue is because I'm reserving it to track replacing the build system to side step the problem.
No id prefer to enable setuptools to provide dynamic requires depending on whether its running on a sdist or on a vcs checkout
But if that fails I'll work towards creating a separate build backend library in the dependency loop of build backends that can self build from its sdist and enable any sdist with matching tooling to do the same without loops
Additionally most users will not need any changes
But any build backend or dependeny of core build backends will change
No id prefer to enable setuptools to provide dynamic requires depending on whether its running on a sdist or on a vcs checkout
But if that fails I'll work towards creating a separate build backend library in the dependency loop of build backends that can self build from its sdist and enable any sdist with matching tooling to do the same without loops
Please bring these ideas to packaging-problems or setuptools; I don't want to continue having packaging ecosystem problems in this issue, as this project is the wrong audience. This issue is about fixing the issue importlib_metadata specifically.
We're presently facing this particular issue with python-3.9.23 and importlib_metadata-8.7.0 :
LookupError: https://files.pythonhosted.org/packages/76/66/650a33bd90f786193e4de4b3ad86ea60b53c89b669a5c7be931fac31cdb0/importlib_metadata-8.7.0.tar.gz (from https://pypi.org/simple/importlib-metadata/) (requires-python:>=3.9) is already being built: importlib-metadata>=4.6 from https://files.pythonhosted.org/packages/76/66/650a33bd90f786193e4de4b3ad86ea60b53c89b669a5c7be931fac31cdb0/importlib_metadata-8.7.0.tar.gz (from setuptools_scm)
white trying to (cross-)compile software stack that includes Python and a few python modules.
https://github.com/pypa/setuptools-scm/pull/1152 should resolve that with the next setuptools_scm release