pypa
Misleading error message for broken setuptools
Description
tl;dr: pip hides the actual ImportError when importing setuptools, hampering diagnosis. Suggested patch below.
A trivial problem in our setup lead into a lengthy search because of missing information. This can happen for legacy packages without build isolation configured via pyproject.toml. Consider this simple setup.py:
from distutils.core import setup
setup(name="hello_pip", version="0.0", py_modules="hello_pip")
Trying to pip install a package like that caused this error with pip:
$ pip install internal-package
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [1 lines of output]
ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
[end of output]
Turns out that in the end, somehow setuptools and jaraco.functools got incompatible in the environment, causing a lengthy error hunt. See below for how to reproduce.
Suggested patch:
diff --git a/src/pip/_internal/utils/setuptools_build.py b/src/pip/_internal/utils/setuptools_build.py
index 96d1b2460..908315593 100644
--- a/src/pip/_internal/utils/setuptools_build.py
+++ b/src/pip/_internal/utils/setuptools_build.py
@@ -21,7 +21,10 @@ _SETUPTOOLS_SHIM = textwrap.dedent(
try:
import setuptools
- except ImportError as error:
+ except ModuleNotFoundError as error:
+ if error.name != "setuptools":
+ raise # some other module failed to load when importing setuptools
+
print(
"ERROR: Can not execute `setup.py` since setuptools is not available in "
"the build environment.",
Other observations
Installing the same package in a fresh Python environment (which does not have setuptools at call) interestingly worked. To reproduce:
FROM python:3.12
RUN pip uninstall -y setuptools
COPY . /src/
RUN pip install /src/
The output misled me to think that our internal package is actually using build isolation and has a pyproject.toml:
#8 [4/4] RUN pip install /src/
#8 0.823 Processing /src
#8 0.826 Installing build dependencies: started
#8 3.963 Installing build dependencies: finished with status 'done'
#8 3.964 Getting requirements to build wheel: started
#8 4.233 Getting requirements to build wheel: finished with status 'done'
#8 4.234 Preparing metadata (pyproject.toml): started
#8 4.491 Preparing metadata (pyproject.toml): finished with status 'done'
#8 4.495 Building wheels for collected packages: hello_pip
#8 4.497 Building wheel for hello_pip (pyproject.toml): started
#8 4.787 Building wheel for hello_pip (pyproject.toml): finished with status 'done'
Expected behavior
pip should expose the actual error that occurred when trying to import setuptools.
pip version
24.2
Python version
3.12.4
OS
Linux
How to Reproduce
$ cat > setup.py <<EOF
from distutils.core import setup
setup(name="hello_pip", version="0.0", py_modules="hello_pip")
EOF
$ cat > hello_pip.py <<EOF
def hello():
return "Hello, pip"
EOF
$ cat > Dockerfile <<EOF
FROM python:3.12
RUN pip install pip==24.2 jaraco.functools==3.7 setuptools==75.1.0
COPY . /src/
RUN pip install /src/
EOF
$ docker build --no-cache --progress=plain .
...
ERROR: failed to solve: process "/bin/sh -c pip install /src/" did not complete successfully: exit code: 1
Output
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 157B done
#1 DONE 0.0s
#2 [internal] load metadata for docker.io/library/python:3.12
#2 DONE 0.0s
#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s
#4 [1/4] FROM docker.io/library/python:3.12
#4 CACHED
#5 [internal] load build context
#5 transferring context: 195B done
#5 DONE 0.0s
#6 [2/4] RUN pip install pip==24.2 jaraco.functools==3.7 setuptools==75.1.0
#6 2.550 Collecting pip==24.2
#6 2.736 Downloading pip-24.2-py3-none-any.whl.metadata (3.6 kB)
#6 2.829 Collecting jaraco.functools==3.7
#6 2.850 Downloading jaraco.functools-3.7.0-py3-none-any.whl.metadata (3.1 kB)
#6 3.322 Collecting setuptools==75.1.0
#6 3.342 Downloading setuptools-75.1.0-py3-none-any.whl.metadata (6.9 kB)
#6 3.450 Collecting more-itertools (from jaraco.functools==3.7)
#6 3.471 Downloading more_itertools-10.5.0-py3-none-any.whl.metadata (36 kB)
#6 3.601 Downloading pip-24.2-py3-none-any.whl (1.8 MB)
#6 5.200 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 1.1 MB/s eta 0:00:00
#6 5.224 Downloading jaraco.functools-3.7.0-py3-none-any.whl (8.1 kB)
#6 5.256 Downloading setuptools-75.1.0-py3-none-any.whl (1.2 MB)
#6 6.355 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 1.2 MB/s eta 0:00:00
#6 6.378 Downloading more_itertools-10.5.0-py3-none-any.whl (60 kB)
#6 6.485 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 kB 603.2 kB/s eta 0:00:00
#6 6.694 Installing collected packages: setuptools, pip, more-itertools, jaraco.functools
#6 6.695 Attempting uninstall: setuptools
#6 6.701 Found existing installation: setuptools 70.0.0
#6 6.734 Uninstalling setuptools-70.0.0:
#6 6.814 Successfully uninstalled setuptools-70.0.0
#6 7.782 Attempting uninstall: pip
#6 7.788 Found existing installation: pip 24.0
#6 7.850 Uninstalling pip-24.0:
#6 8.056 Successfully uninstalled pip-24.0
#6 9.248 Successfully installed jaraco.functools-3.7.0 more-itertools-10.5.0 pip-24.2 setuptools-75.1.0
#6 9.248 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#6 DONE 9.8s
#7 [3/4] COPY . /src/
#7 DONE 0.0s
#8 [4/4] RUN pip install /src/
#8 0.744 Processing /src
#8 0.746 Preparing metadata (setup.py): started
#8 0.825 Preparing metadata (setup.py): finished with status 'error'
#8 0.832 error: subprocess-exited-with-error
#8 0.832
#8 0.832 × python setup.py egg_info did not run successfully.
#8 0.832 │ exit code: 1
#8 0.832 ╰─> [1 lines of output]
#8 0.832 ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
#8 0.832 [end of output]
#8 0.832
#8 0.832 note: This error originates from a subprocess, and is likely not a problem with pip.
#8 0.877 error: metadata-generation-failed
#8 0.877
#8 0.877 × Encountered error while generating package metadata.
#8 0.877 ╰─> See above for output.
#8 0.877
#8 0.877 note: This is an issue with the package mentioned above, not pip.
#8 0.877 hint: See above for details.
#8 ERROR: process "/bin/sh -c pip install /src/" did not complete successfully: exit code: 1
------
> [4/4] RUN pip install /src/:
0.832 [end of output]
0.832
0.832 note: This error originates from a subprocess, and is likely not a problem with pip.
0.877 error: metadata-generation-failed
0.877
0.877 × Encountered error while generating package metadata.
0.877 ╰─> See above for output.
0.877
0.877 note: This is an issue with the package mentioned above, not pip.
0.877 hint: See above for details.
------
Dockerfile:4
--------------------
2 | RUN pip install pip==24.2 jaraco.functools==3.7 setuptools==75.1.0
3 | COPY . /src/
4 | >>> RUN pip install /src/
5 |
6 |
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install /src/" did not complete successfully: exit code: 1
Code of Conduct
- [X] I agree to follow the PSF Code of Conduct.
@pfmoore Obviously it can't because the information was already eaten in the _SETUPTOOLS_SHIM. I also tried using the verbose output of pip when initially digging into this, to no avail.
Here is the output with -vvv:
#8 0.791 Running command python setup.py egg_info
#8 0.863 ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
#8 0.882 error: subprocess-exited-with-error
#8 0.882
#8 0.882 × python setup.py egg_info did not run successfully.
#8 0.882 │ exit code: 1
#8 0.882 ╰─> See above for output.
#8 0.882
#8 0.882 note: This error originates from a subprocess, and is likely not a problem with pip.
#8 0.884 full command: /usr/local/bin/python -c '
#8 0.884 exec(compile('"'"''"'"''"'"'
#8 0.884 # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
#8 0.884 #
#8 0.884 # - It imports setuptools before invoking setup.py, to enable projects that directly
#8 0.884 # import from `distutils.core` to work with newer packaging standards.
#8 0.884 # - It provides a clear error message when setuptools is not installed.
#8 0.884 # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
#8 0.884 # setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
#8 0.884 # manifest_maker: standard file '"'"'-c'"'"' not found".
#8 0.884 # - It generates a shim setup.py, for handling setup.cfg-only projects.
#8 0.884 import os, sys, tokenize
#8 0.884
#8 0.884 try:
#8 0.884 import setuptools
#8 0.884 except ImportError as error:
#8 0.884 print(
#8 0.884 "ERROR: Can not execute `setup.py` since setuptools is not available in "
#8 0.884 "the build environment.",
#8 0.884 file=sys.stderr,
#8 0.884 )
#8 0.884 sys.exit(1)
#8 0.884
#8 0.884 __file__ = %r
#8 0.884 sys.argv[0] = __file__
#8 0.884
#8 0.884 if os.path.exists(__file__):
#8 0.884 filename = __file__
#8 0.884 with tokenize.open(__file__) as f:
#8 0.884 setup_py_code = f.read()
#8 0.884 else:
#8 0.884 filename = "<auto-generated setuptools caller>"
#8 0.884 setup_py_code = "from setuptools import setup; setup()"
#8 0.884
#8 0.884 exec(compile(setup_py_code, filename, "exec"))
#8 0.884 '"'"''"'"''"'"' % ('"'"'/src/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' egg_info --egg-base /tmp/pip-pip-egg-info-9743iszq
#8 0.884 cwd: /src/
#8 0.885 Preparing metadata (setup.py): finished with status 'error'
#8 0.929 Remote version of pip: 24.2
#8 0.930 Local version of pip: 24.2
#8 0.934 Was pip installed by pip? True
#8 0.938 error: metadata-generation-failed
#8 0.938
#8 0.938 × Encountered error while generating package metadata.
#8 0.938 ╰─> See above for output.
#8 0.938
#8 0.938 note: This is an issue with the package mentioned above, not pip.
#8 0.938 hint: See above for details.
#8 0.945 Exception information:
#8 0.945 Traceback (most recent call last):
#8 0.945 File "/usr/local/lib/python3.12/site-packages/pip/_internal/operations/build/metadata_legacy.py", line 64, in generate_metadata
#8 0.945 call_subprocess(
#8 0.945 File "/usr/local/lib/python3.12/site-packages/pip/_internal/utils/subprocess.py", line 209, in call_subprocess
#8 0.945 raise error
#8 0.945 pip._internal.exceptions.InstallationSubprocessError: python setup.py egg_info exited with 1
#8 0.945
#8 0.945 The above exception was the direct cause of the following exception:
...
Ah, wait. This is in the legacy build path that uses setuptools when there's no pyproject.toml. I'd suggest you add a pyproject.toml to your project (specifying setuptools as the build backend). The handling (and reporting) of errors should be better in that case.
We're unlikely to do much with the legacy code path at this point, as it's been deprecated for some time.
Actually that is what we did: Add a pyproject.toml. This fixed it for one package, but we have like 30 that are old enough to break in this way. :cry:
Still trying to find a way out of this mess, especially for allowing point releases on old release branches.
Close this if you like, my main goal was to document what might be going on here. Anyway, I think applying the trivial patch can not hurt. YMMV
Thanks. I guess the answer here is that if you don't have the resources to update your packages, you should probably be cautious about upgrading pip, as you'll find that newer versions of pip will gradually stop working with out of date packaging practices.
One thing you could try is --use-pep517. That forces the use of the newer code path without requiring you to add a pyproject.toml. It's something of a stopgap, as it was only ever intended as a transitional option, but it might get you out of trouble for now, at least.
Thanks for the pointer. Actually, not pip was the problem here but an updated setuptools, that actually requires a newer version of jaraco.functools.
I only created the issue here because I ran in the wrong direction due to the incomplete error information.
Kudos, Torsten