pypa
implicit dependency on setuptools causes build failures on pip
In pypa/pip#7831, I described an issue where attempting to build distlib with pep-571 support and --no-binary :all: leads to a build failure when pip attempts to create the build environment with setuptools and wheel, but wheel fails to build when setuptools isn't present. Perhaps this issue could be addressed by declaring the build dependency on setuptools.
Repro:
$ pip-run --no-binary :all: wheel
Collecting wheel
Using cached wheel-0.34.2.tar.gz (58 kB)
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-kqex__en/wheel/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-kqex__en/wheel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-kqex__en/wheel/pip-egg-info
cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-kqex__en/wheel/
Complete output (3 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pip-run", line 8, in <module>
sys.exit(run())
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/__init__.py", line 18, in run
with deps.load(*deps.not_installed(pip_args)) as home:
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/deps.py", line 75, in load
_installable(args) and subprocess.check_call(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-uz4z12c7', '--no-binary', ':all:', 'wheel')' returned non-zero exit status 1.
In 4f9d93d, I attempted to declare the dependency on setuptools, but that was insufficient. It seems pip was unable to honor it.
$ pip-run --no-binary :all: git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools
Running command git clone -q https://github.com/pypa/wheel /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-d31ewf1h
Collecting git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools
Cloning https://github.com/pypa/wheel (to revision bugfix/344-explicit-setuptools) to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-d31ewf1h
Running command git checkout -b bugfix/344-explicit-setuptools --track origin/bugfix/344-explicit-setuptools
Switched to a new branch 'bugfix/344-explicit-setuptools'
Branch 'bugfix/344-explicit-setuptools' set up to track remote branch 'bugfix/344-explicit-setuptools' from 'origin'.
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-p3mtk_lh/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- wheel
cwd: None
Complete output (11 lines):
Collecting wheel
Using cached wheel-0.34.2.tar.gz (58 kB)
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/pip-egg-info
cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/
Complete output (3 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-p3mtk_lh/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- wheel Check the logs for full command output.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pip-run", line 8, in <module>
sys.exit(run())
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/__init__.py", line 18, in run
with deps.load(*deps.not_installed(pip_args)) as home:
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/deps.py", line 75, in load
_installable(args) and subprocess.check_call(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-trtwrd5a', '--no-binary', ':all:', 'git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools')' returned non-zero exit status 1.
It seems pip is still trying to install wheel, even though it's not in the build-system.requires.
Edit: To replicate this behavior use pip-run --no-binary :all: git+https://github.com/pypa/wheel@4f9d93d.
In cae8be50488b007cbd6c261a86ca478b473631a9, I tried adding backend-path so that wheel had wheel, but that fails with a BackendInvalid error:
$ pip-run --no-binary :all: git+https://github.com/pypa/wheel@cae8be50488b007cbd6c261a86ca478b473631a9
Collecting git+https://github.com/pypa/wheel@cae8be50488b007cbd6c261a86ca478b473631a9
Cloning https://github.com/pypa/wheel (to revision cae8be50488b007cbd6c261a86ca478b473631a9) to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-i4zvhvwd
Running command git clone -q https://github.com/pypa/wheel /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-i4zvhvwd
Running command git checkout -q cae8be50488b007cbd6c261a86ca478b473631a9
Installing build dependencies ... done
Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
status = self.run(options, args)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 331, in run
resolver.resolve(requirement_set)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
discovered_reqs.extend(self._resolve_one(requirement_set, req))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
abstract_dist = self.preparer.prepare_linked_requirement(req)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 515, in prepare_linked_requirement
abstract_dist = _get_prepared_distribution(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(finder, build_isolation)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
self._setup_isolation(finder)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 96, in _setup_isolation
reqs = backend.get_requires_for_build_wheel()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 151, in get_requires_for_build_wheel
return self._call_hook('get_requires_for_build_wheel', {
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 257, in _call_hook
raise BackendInvalid(
pip._vendor.pep517.wrappers.BackendInvalid
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pip-run", line 8, in <module>
sys.exit(run())
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/__init__.py", line 18, in run
with deps.load(*deps.not_installed(pip_args)) as home:
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/deps.py", line 75, in load
_installable(args) and subprocess.check_call(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ql9o180s', '--no-binary', ':all:', 'git+https://github.com/pypa/wheel@cae8be50488b007cbd6c261a86ca478b473631a9')' returned non-zero exit status 2.
What do you expect me to do here? Wheel already declares setuptools as a setup requirement. Creating a pyproject.toml has been tried too and its presence caused other issues.
I'm not sure there's anything the wheel project can do here. It seems to be a bootstrapping issue - pip creates an implicit dependency on wheel for every project, including setuptools, so when pip attempts to build wheel from source, it attempts to download setuptools, which requires wheel (still from source), which can't build without setuptools. This issue probably needs to be addressed at the pip or packaging-problems level. The aforementioned pip issue is probably a suitable forum for the underlying issue.
Creating a
pyproject.tomlhas been tried too and its presence caused other issues.
That seems problemmatic. Even if supplying a pyproject.toml doesn't address the bootstrapping issue, it shouldn't be the case that following PEP 517/518 should cause additional failures. Do you have any references as to what issues that caused?
Seems like one possibility towards bootstrapping solution here would be porting this project to be built through flit. As part of testing https://github.com/takluyver/flit/pull/441, I noticed that a simple bootstrapper PoC like https://gitlab.com/freedesktop-sdk/pep517_install/-/tree/main would be able to install as far as flit itself and could be used to install this project which would in term allow installing setuptools (whose build dep is wheel).
I've been working on solving the boostrapping problem in general in https://github.com/pypa/packaging-problems/issues/342. I'm closing this as won't fix. I've pushed the work in progress branch to refs/archive/bugfix/344-explicit-setuptools and deleted the branch.
You actually closed this as completed, not wontfix. But I think this is actually correct, as wheel no longer depends on setuptools to install, given that it uses flit instead to overcome this specific problem. There is currently a run-time dependency on setuptools, for wheel convert, but I'm also working to remove even that dependency by refactoring the code.