pip-25.0.1 runs setup.py with broken import behaviour

[julian-smith-artifex-com]

Description

pip-25.0.1 appears to run setup.py in such a way that import platform picks up a local platform/ directory if it exists, instead of the built-in module.

This happens if platform/ is empty, or if it contains other files and directories without a __init__.py file.

I include a Python script that shows the broken behaviour.

• Run with no args: ./pipbug.py.
• If not run in a venv it reruns itself inside a new venv.
• Then it creates a new project directory with a pyproject.toml file, a (non-working) setup.py and an empty platform/ directory.
• Then it attempts to build a wheel with pip-24.3.1 and pip-25.0.1.

Expected behavior

import platform should get the built-in module regardless of whether there is a local platform directory.

pip version

25.0.1

Python version

3.11 and 3.12.

OS

Linux, OpenBSD.

How to Reproduce

Run this Python script as file pipbug.py:

#!/usr/bin/env python3

'''
pip-25.0.1 runs setup.py in such a way that `import platform` picks up local
`platform/` directory, not the built-in module.
'''

import os
import shlex
import shutil
import subprocess
import sys
import textwrap


filep = os.path.normpath(__file__)

def run(command, check=1):
    print(f'Running: {command}', flush=1)
    return subprocess.run(command, shell=1, check=check)

def main():
    # Create project directory {filep}_testdir
    testdir = f'{filep}_testdir'
    shutil.rmtree(testdir, ignore_errors=1)
    os.mkdir(testdir)
    # Create empty `platform/` directory.
    os.mkdir(f'{testdir}/platform')
    
    # Create (non-working) setup.py.
    with open(f'{testdir}/setup.py', 'w') as f:
        f.write(
                textwrap.dedent('''
                    import os
                    import platform
                    import sys
                    print(f'setup.py: {sys.path=}')
                    print(f'setup.py: {os.getcwd()=}')
                    print(f'setup.py: {dir(platform)=}')
                    print(f'setup.py: {platform.__file__=}')
                    print(f'setup.py: {getattr(platform, "__path__", None)=}')
                    print(f'setup.py: {platform.system()=}')
                    ''')
                )

    # Create pyproject.toml.
    with open(f'{testdir}/pyproject.toml', 'w') as f:
        f.write(
                textwrap.dedent('''
                    [build-system]
                    requires = []

                    # See pep-517.
                    #
                    build-backend = "setup"
                    backend-path = ["."]
                    ''')
                )

    # Attempt to create wheel with pip-24.3.1. This gives expected error:
    #   AttributeError: module 'setup' has no attribute 'build_wheel'
    print('=' * 80)
    print(f'Testing with pip-24.3.1')
    run(f'pip install pip==24.3.1')
    run(f'pip wheel -v {os.path.abspath(testdir)}', check=0)
    
    # Attempt to create wheel with pip-25.0.1. This gives different error:
    #   AttributeError: module 'platform' has no attribute 'system'
    print('=' * 80)
    print(f'Testing with pip-25.0.1')
    run(f'pip install pip==25.0.1')
    run(f'pip wheel -v {os.path.abspath(testdir)}', check=0)


if __name__ == '__main__':
    if sys.prefix == sys.base_prefix:
        # Not running in a venv. Rerun ourselves in a venv.
        command = f'{sys.executable} -m venv {filep}_venv'
        command += f' && . {filep}_venv/bin/activate'
        command += f' && python'
        for arg in sys.argv:
            command += f' {shlex.quote(arg)}'
        run(command)
    else:
        main()

Output

Example output on Linux is below.

• With pip-24.3.1 we get expected error `AttributeError: module 'setup' has no attribute 'build_wheel'.
• With pip-25.0.1 we get earlier error AttributeError: module 'platform' has no attribute 'system', because import platform has picked up the local empty platform directory instead of the built-in module platform.

> ./pipbug.py
Running: /usr/bin/python3 -m venv [...]/pipbug.py_venv && . [...]/pipbug.py_venv/bin/activate && python ./pipbug.py
================================================================================
Testing with pip-24.3.1
Running: pip install pip==24.3.1
Collecting pip==24.3.1
  Using cached pip-24.3.1-py3-none-any.whl.metadata (3.7 kB)
Using cached pip-24.3.1-py3-none-any.whl (1.8 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 25.0.1
    Uninstalling pip-25.0.1:
      Successfully uninstalled pip-25.0.1
Successfully installed pip-24.3.1

[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
Running: pip wheel -v [...]/pipbug.py_testdir
Processing ./pipbug.py_testdir
  Getting requirements to build wheel: started
  Running command Getting requirements to build wheel
  setup.py: sys.path=['[...]/pipbug.py_testdir', '[...]/pipbug.py_venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process', '/tmp/pip-build-env-4awji4kl/site', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/tmp/pip-build-env-4awji4kl/overlay/lib/python3.12/site-packages', '/tmp/pip-build-env-4awji4kl/normal/lib/python3.12/site-packages']
  setup.py: os.getcwd()='[...]/pipbug.py_testdir'
  setup.py: dir(platform)=['_Processor', '_WIN32_CLIENT_RELEASES', '_WIN32_SERVER_RELEASES', '__builtins__', '__cached__', '__copyright__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '__version__', '_comparable_version', '_default_architecture', '_follow_symlinks', '_get_machine_win32', '_java_getprop', '_mac_ver_xml', '_node', '_norm_version', '_os_release_cache', '_os_release_candidates', '_parse_os_release', '_platform', '_platform_cache', '_sys_version', '_sys_version_cache', '_syscmd_file', '_syscmd_ver', '_uname_cache', '_unknown_as_blank', '_ver_stages', '_win32_ver', '_wmi_query', 'architecture', 'collections', 'freedesktop_os_release', 'functools', 'itertools', 'java_ver', 'libc_ver', 'mac_ver', 'machine', 'node', 'os', 'platform', 'processor', 'python_branch', 'python_build', 'python_compiler', 'python_implementation', 'python_revision', 'python_version', 'python_version_tuple', 're', 'release', 'sys', 'system', 'system_alias', 'uname', 'uname_result', 'version', 'win32_edition', 'win32_is_iot', 'win32_ver']
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  setup.py: platform.__file__='/usr/lib/python3.12/platform.py'
  setup.py: getattr(platform, "__path__", None)=None
  setup.py: platform.system()='Linux'
  Running command Preparing metadata (pyproject.toml)
  setup.py: sys.path=['[...]/pipbug.py_testdir', '[...]/pipbug.py_venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process', '/tmp/pip-build-env-4awji4kl/site', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/tmp/pip-build-env-4awji4kl/overlay/lib/python3.12/site-packages', '/tmp/pip-build-env-4awji4kl/normal/lib/python3.12/site-packages']
  setup.py: os.getcwd()='[...]/pipbug.py_testdir'
  setup.py: dir(platform)=['_Processor', '_WIN32_CLIENT_RELEASES', '_WIN32_SERVER_RELEASES', '__builtins__', '__cached__', '__copyright__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '__version__', '_comparable_version', '_default_architecture', '_follow_symlinks', '_get_machine_win32', '_java_getprop', '_mac_ver_xml', '_node', '_norm_version', '_os_release_cache', '_os_release_candidates', '_parse_os_release', '_platform', '_platform_cache', '_sys_version', '_sys_version_cache', '_syscmd_file', '_syscmd_ver', '_uname_cache', '_unknown_as_blank', '_ver_stages', '_win32_ver', '_wmi_query', 'architecture', 'collections', 'freedesktop_os_release', 'functools', 'itertools', 'java_ver', 'libc_ver', 'mac_ver', 'machine', 'node', 'os', 'platform', 'processor', 'python_branch', 'python_build', 'python_compiler', 'python_implementation', 'python_revision', 'python_version', 'python_version_tuple', 're', 'release', 'sys', 'system', 'system_alias', 'uname', 'uname_result', 'version', 'win32_edition', 'win32_is_iot', 'win32_ver']
  setup.py: platform.__file__='/usr/lib/python3.12/platform.py'
  setup.py: getattr(platform, "__path__", None)=None
  setup.py: platform.system()='Linux'
  Traceback (most recent call last):
    File "[...]/pipbug.py_venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "[...]/pipbug.py_venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "[...]/pipbug.py_venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
      whl_basename = backend.build_wheel(metadata_directory, config_settings)
                     ^^^^^^^^^^^^^^^^^^^
  AttributeError: module 'setup' has no attribute 'build_wheel'
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: [...]/pipbug.py_venv/bin/python3 [...]/pipbug.py_venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpoiz5wz0s
  cwd: [...]/pipbug.py_testdir

[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
================================================================================
Testing with pip-25.0.1
Running: pip install pip==25.0.1
Collecting pip==25.0.1
  Using cached pip-25.0.1-py3-none-any.whl.metadata (3.7 kB)
Using cached pip-25.0.1-py3-none-any.whl (1.8 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.3.1
    Uninstalling pip-24.3.1:
      Successfully uninstalled pip-24.3.1
Successfully installed pip-25.0.1
Running: pip wheel -v [...]/pipbug.py_testdir
Processing ./pipbug.py_testdir
  Getting requirements to build wheel: started
  Running command Getting requirements to build wheel
  setup.py: sys.path=['/tmp/pip-build-env-9swhkfon/site', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/tmp/pip-build-env-9swhkfon/overlay/lib/python3.12/site-packages', '/tmp/pip-build-env-9swhkfon/normal/lib/python3.12/site-packages']
  setup.py: os.getcwd()='[...]/pipbug.py_testdir'
  setup.py: dir(platform)=['__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
  setup.py: platform.__file__=None
  setup.py: getattr(platform, "__path__", None)=_NamespacePath(['[...]/pipbug.py_testdir/platform'])
  Traceback (most recent call last):
    File "[...]/pipbug.py_venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
      main()
    File "[...]/pipbug.py_venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
      json_out["return_val"] = hook(**hook_input["kwargs"])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "[...]/pipbug.py_venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 137, in get_requires_for_build_wheel
      backend = _build_backend()
                ^^^^^^^^^^^^^^^^
    File "[...]/pipbug.py_venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 70, in _build_backend
      obj = import_module(mod_path)
            ^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 999, in exec_module
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "[...]/pipbug.py_testdir/setup.py", line 10, in <module>
      print(f'setup.py: {platform.system()=}')
                         ^^^^^^^^^^^^^^^
  AttributeError: module 'platform' has no attribute 'system'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: [...]/pipbug.py_venv/bin/python3 [...]/pipbug.py_venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp1lbpiwek
  cwd: [...]/pipbug.py_testdir
  Getting requirements to build wheel: finished with status 'error'
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Code of Conduct

• [x] I agree to follow the PSF Code of Conduct.

[julian-smith-artifex-com]

The same problem also occurs on Windows (needs some minor modifications to the script).

[notatallshaw]

pip-25.0.1 appears to run setup.py in such a way that import platform picks up a local platform/ directory if it exists, instead of the built-in module.

Isn't this standard Python behavior?

If you create a directory or Python file in the same directory, then it becomes importable, and has higher precedence than standard library modules.

What else would you expect? What if someone was importing a local module with that name?

[julian-smith-artifex-com]

I don't think it is standard Python behaviour - i think a local directory needs to have a __init__.py file in order to be considered for import.

Without a __init__,py file, i think a local directory should not be considered for import. This is how plain Python behaves, and how setup.py behaves when invoked by older pip versions.

BTW i can't provide exact references to PEPs etc for this, import is pretty complicated and i don't pretend to understand all the details. But i do know that pip-25.0.1 has changed how setup's import statements behave.

[pfmoore]

I don't think it is standard Python behaviour - i think a local directory needs to have a __init__.py file in order to be considered for import.

No, see PEP 420 - Implicit Namespace Packages. Plain directories have been importable since Python 3.3.

[RonnyPfannschmidt]

could https://github.com/pypa/pyproject-hooks/pull/199 be related

i suspect it may start to identify the local platform directory as viable namespace package with that but i haven't verified that idea

[RonnyPfannschmidt]

i suspect this may be a bug in behaviour of the meta path finder in pyproject hook

i believe its expected to try if another path hook find this as a non-namespace package before designating it as such

[RonnyPfannschmidt]

https://github.com/pypa/pyproject-hooks/issues/207 looks like a very matching explanation

i beleive the reproducer can be turned into a regression test for pyproject hooks

[julian-smith-artifex-com]

I don't think it is standard Python behaviour - i think a local directory needs to have a __init__.py file in order to be considered for import.

No, see PEP 420 - Implicit Namespace Packages. Plain directories have been importable since Python 3.3.

Ah, apologies, i was completely unaware of this.

[But presumably there's still a problem with pip-25.0.1 because the built-in platform module should always be preferred over a bare platform/ directory?]

[hroncok]

About namespace packages: creating an empty platform directory in a directory where a script exists and executing the script with import platform in it imports the standard library platform module.

[mjg]

Is there something going on beyond messing with the path? If I put

#!/usr/bin/env python3

''' Work around pip 25/pyproject_hooks 1.2.0 path meddling '''

from platform import *

into platform/__init__.py (in the source tree, not sys) then nothing changes, whereas I would expect either

• platform to have all attributes of platform (huh) or
• python to complain about a circular import.

That is, unless python silently discards a circular import.

[mjg]

It gets even more interesting. Even if

• I pop "platform" from sys.modules before the import
• and sys.path does not contain the parent dir of the platform dir nor that dir

the import platform still identifies the dir platform as a namespace (when called via the setup hooks). So, maybe it's something that pip/pyproject_hooks does to sys.meta_path or importlib?

[mjg]

So it seems to be pyproject_hooks injecting its own Finder into sys.meta_path. I don't know what the proper fix is in pip (not doing that, removing the Finder, fixing the Finder), but I know how to work around. PR for mupdf and FTBFS fix for Fedora package coming right up in the respective places.

[notatallshaw]

In pip 25.0 pyproject-hooks was upgraded from 1.0.0 to 1.2.0, so if it was a consequence of a change from pyproject-hooks then it should be one of these PRs: https://github.com/pypa/pyproject-hooks/pulls?q=is%3Apr+is%3Amerged+merged%3A2022-11-22..2024-09-29

To me, the obvious candidates are https://github.com/pypa/pyproject-hooks/pull/165 and the follow up PR https://github.com/pypa/pyproject-hooks/pull/193

[mjg]

AFAICT it is commit 084b02e in pyprojetc-hooks which looks like a big squash of several commits. It is beyond me why people do that - it prevents isolated reverts, let alone bisecting. In pip, you do not use submodules but import a source dump of vendored modules (I know why) in a commit which contains other adjustments also. Again ... This makes it difficult for me to work on and suggest a change, blame it on my incompatibility with this kind of git usage, sorry. All I can tell you is that on the pip consumer side, I work around like this:

https://github.com/ArtifexSoftware/mupdf/pull/68/commits/8c43f4fa381a8f9e19506b6fd9a1316eec53ed11

It's the last commit in a PR which - originally - consisted of 1 commit only but got rebased/rewritten meanwhile, it seems. So it clearly is the _BackendPathFinder in the changed sys.meta_path which inserts those namespaces. Cleaning up sys.meta_path helps, as would fixing the finder (I don't know how, I've tried).

[oerp-odoo]

I've recently run into this problem as well. In our docker container, we have a directory /opt/odoo and then we have odoo package that is installed using -e option. But now if you for example enter python shell when your working directory is /opt/odoo, it will overwrite odoo package and any import will fail. It was working fine previously.

[sbidoul]

@oerp-odoo Odoo has specific issues (for instance it requires --config-setting=editable_mode=compat). I'd recommend discussing Odoo specifics in https://github.com/odoo/odoo/pull/44001 first (I'll be happy to help there), and bring it back here if we conclude there is a a real issue in pip.

[oerp-odoo]

@sbidoul turns out the real issue is not pip, but setuptools. After setuptools was autoupdated to >=78.1.0, it started breaking with editable packages. If I downgrade it to lower version, it starts working correctly again: https://github.com/pypa/setuptools/issues/4943