Pip 22.1 (only) fails when included in pyproject.toml

[cowlinator]

Description

Starting with pip 22.1, including "pip" in the pyproject.toml [build-system] requires fails.

Example pyproject.toml:

[build-system]
requires = [
    "setuptools>=40.8.0", 
    "wheel",
    "pip"
]
build-backend = "setuptools.build_meta"

Error:

(venv) D:\Dev\myPyPackage>python -m pip install D:\Dev\myPyPackage
Looking in indexes: https://pypi.org/simple
Processing d:\dev\mypypackage
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      Looking in indexes: https://pypi.org/simple
      Collecting setuptools>=40.8.0
        Using cached setuptools-62.2.0-py3-none-any.whl (1.1 MB)
      Collecting wheel
        Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
      Collecting pip
        Using cached pip-22.1-py3-none-any.whl (2.1 MB)
      ERROR: To modify pip, please run the following command:
      D:\Dev\myPyPackage\venv\Scripts\python.exe -m pip install --ignore-installed --no-user --prefix C:\Users\cowlinator\AppData\Local\Temp\pip-build-env-upw4u049\overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --trusted-host pypi.org -- setuptools>=40.8.0 wheel pip
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Verified that this succeeds with pip 22.0.4

This error message repeatedly states that this is likely not a problem with pip... but it is the change from pip 22.0.4 to 22.1 that breaks it.

Expected behavior

I expect pip 22.1 to behave the same as pip 22.0.4; namely, that the installation succeeds.

Output when using pip 22.0.4:

(venv) D:\Dev\myPyPackage>python -m pip install D:\Dev\myPyPackage
Looking in indexes: https://pypi.org/simple
Processing d:\dev\mypypackage
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: hi
  Building wheel for hi (pyproject.toml) ... done
  Created wheel for hi: filename=hi-0.0.0-py3-none-any.whl size=1048 sha256=b5e4d2b574164c3e8c93ea425bd63f58876eb58087dedd259755a9326ddc84fd
  Stored in directory: C:\Users\cowlinator\AppData\Local\Temp\pip-ephem-wheel-cache-u03_8ou9\wheels\6a\64\7b\c6556dc063c160509a71a3f427d795bea205706ceccf1319ff
Successfully built hi
Installing collected packages: hi
Successfully installed hi-0.0.0
WARNING: You are using pip version 22.0.4; however, version 22.1 is available.
You should consider upgrading via the 'D:\Dev\myPyPackage\venv\Scripts\python.exe -m pip install --upgrade pip' command.

pip version

22.1

Python version

3.7.9

OS

Windows 11 (21H2)

How to Reproduce

1. Create a virtual environment and activate it
2. Install pip 22.1
3. Create a python package with a pyproject.toml containing:

[build-system]
requires = [
    "setuptools>=40.8.0", 
    "wheel",
    "pip"
]
build-backend = "setuptools.build_meta"

4. Install this python package from the local directory with python -m pip install <dir>
5. Observe the error

Output

D:\Dev\myPyPackage>python -m venv venv && venv\Scripts\activate

(venv) D:\Dev\myPyPackage>python -m pip install --upgrade pip
Looking in indexes: https://pypi.org/simple
Collecting pip
  Using cached pip-22.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.1.1
    Uninstalling pip-20.1.1:
      Successfully uninstalled pip-20.1.1
Successfully installed pip-22.1

(venv) D:\Dev\myPyPackage>python -m pip install D:\Dev\myPyPackage
Looking in indexes: https://pypi.org/simple
Processing d:\dev\mypypackage
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      Looking in indexes: https://pypi.org/simple
      Collecting setuptools>=40.8.0
        Using cached setuptools-62.2.0-py3-none-any.whl (1.1 MB)
      Collecting wheel
        Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
      Collecting pip
        Using cached pip-22.1-py3-none-any.whl (2.1 MB)
      ERROR: To modify pip, please run the following command:
      D:\Dev\myPyPackage\venv\Scripts\python.exe -m pip install --ignore-installed --no-user --prefix C:\Users\cowlinator\AppData\Local\Temp\pip-build-env-upw4u049\overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --trusted-host pypi.org -- setuptools>=40.8.0 wheel pip
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Code of Conduct

• [X] I agree to follow the PSF Code of Conduct.

[pfmoore]

You shouldn't have pip in build-requires in the first place - why do you think you need it there?

The reason it's failing is a somewhat complex combination of the fact that pip is already in the build environment to install everything else, pip can't upgrade itself on Windows unless it's invoked via python -m pip, and when setting up the build environment it looks like we invoke pip rather than python -m pip. It's probably fixable, but messy to do so.

I don't think this is worth fixing because you shouldn't have pip in build-requires in the first place.

[pradyunsg]

This might be something that we could prevent -- I don't have a Windows machine to be able to see why/what is happening. What's happening here is that the environment is basically triggering the logic meant to protect users from a broken pip installation on Windows (see https://github.com/pypa/pip/issues/1299).

It's unclear to me why that's happening, but we might have a spot where code needs to change from pip ... to sys.executable -m pip ... in the codebase -- or some edge case that needs to be accounted for?

If someone is able to reproduce this, please plug a debugger into pip and share the values in the locals() and sys.argv[0] at:

https://github.com/pypa/pip/blob/a9d4446dc4bdfdda803580ba6d8099a674a138fd/src/pip/_internal/utils/misc.py#L736

[pradyunsg]

This error message repeatedly states that this is likely not a problem with pip...

Yea, this happening in a build step, where the typical failure is things like the package not having the appropriate build-time dependencies installed. The use of the word likely points toward that. :)

[pfmoore]

@pradyunsg this is indeed probably the solution, although we may be invoking the build environment's pip at this point, so sys.executable might be the wrong thing to use.

As I say, it's possible, but tricky, and I'd like to know why the OP wants to include pip in build-system.requires before we "fix" this.

[pradyunsg]

Bah, you're right -- I missed that nuance. :)

[cowlinator]

You shouldn't have pip in build-requires in the first place - why do you think you need it there?

...I had no reason to think it didn't need to be there? It needs setuptools and wheel to be there. Why do those need to be there and not pip? The documentation states that build-system.requires is a list of requirement specifiers for build-time dependencies of a package. It seems like pip is a build-time dependency of my package. I don't see how it can pip install if the user doesn't even have pip.

If no pyproject.toml should ever have pip in build-system.requires, then the documentation does not make this clear.

[pfmoore]

If no pyproject.toml should ever have pip in build-system.requires, then the documentation does not make this clear.

build-system.requires is intended to be "the things you need in the environment order to build the project". It's the front end's responsibility to install those items into the environment and to call the build_wheel hook of the backend in that environment. That build frontend might be pip or it might be something else. But it doesn't need to be in the environment because it orchestrates the creation and use of that environment.

I don't think it ever occurred to me that people might think pip needed to be added into the build environment. If you were building a wheel using build would you expect to put build in there? How can you know what frontend the user is invoking? But if you have a suggestion for improving the documentation, that would be great! I guess the first question is which documentation you'd want to improve - this stuff is generally covered in packaging.python.org, not in pip's documentation, so a PR there might be the best approach.

[pradyunsg]

This is covered in pip's documentation, as part of https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-time-dependencies, which might be the sentence that is the source of the confusion here.

[pradyunsg]

Is this still happening in 22.1.1?

[agronholm]

...I had no reason to think it didn't need to be there? It needs setuptools and wheel to be there

Why does wheel need to be there? Setuptools lists it as a dependency for building wheels and is automatically installed when necessary.

[pradyunsg]

It used to not do that, in earlier versions of setuptools.

[grgmiller]

hi @pradyunsg it appears that this is still an issue in 22.1.2. I've been experiencing the same issue originally described in this issue when trying to create environments with packages that include a pip requirement in their pyproject.toml file. See the above linked issues.

[pradyunsg]

Hi @grgmiller! Could you share the details of locals, as I've requested for in https://github.com/pypa/pip/issues/11120#issuecomment-1126684654

I haven't been able to reproduce this myself, so having someone else do it and share details, to diagnose this and look into this, would be helpful!

[grgmiller]

I'm not super familiar with debugging third party code... Can you share any guidance on what you mean by plugging a debugger into pip and accessing those variables? I was getting these errors when when setting up an environment in conda, so not sure how I would access these variables as part of that process.

[pradyunsg]

There's a few ways but, the easiest would be...

• Run python -c "import pip; print(pip)".
• That'll print a directory, navigate to it, and get to pip/_internal/utils/misc.py
• Add a print(locals()) above the should_show_use_python_msg assignment (around line 736).
• Run a pip command that triggers this issue, in a terminal window. The pip command should print the contents of that locals there.
• You should see the locals printed. Post that here!
• Remove the line that you added. If you're super paranoid, you can do pip install --force-reinstall pip instead, which will reinstall pip and remove any modifications. :)

[grgmiller]

Thanks for the help. Here's the output:

{'modifying_pip': True, 'pip_names': ['pip', 'pip3', 'pip3.10']}

[pradyunsg]

And, at the cost of being slightly annoying, what's sys.argv[0] at that point?

[grgmiller]

sys.argv[0] is C:\Users\Greg\AppData\Local\Temp\pip-standalone-pip-r8sq_f5z\__env_pip__.zip\pip

[pradyunsg]

AH, I see what's happening now. Thanks for sharing this @grgmiller!

[pradyunsg]

This is resolved with pip 22.2, I believe.