Warn when two packages write to the same module

[konstin]

We regularly get confusing bug reports where a package sometimes works and sometimes doesn't and it's not clear to the user why. Ultimately, it turns out that two packages contain the same module and there is a race condition when installing the two packages. Usually, it's one of the opencv-python distributions, but recently it's been z3, too. These error are completely inscrutable to users.

• https://github.com/astral-sh/uv/issues/10708
• https://github.com/astral-sh/uv/issues/11806
• https://github.com/astral-sh/uv/issues/11659
• https://github.com/astral-sh/uv/issues/13435

We now warn for top-level modules (pattern: <identifier>/__init__.py) that collide in a single installation, naming the offending wheels.

Test script:

uv venv -q && cargo run -q --profile fast-build pip install --no-progress --link-mode clone opencv-python opencv-contrib-python --no-build --no-deps
uv venv -q && cargo run -q --profile fast-build pip install --no-progress --link-mode copy opencv-python opencv-contrib-python --no-build --no-deps
uv venv -q && cargo run -q --profile fast-build pip install --no-progress --link-mode hardlink opencv-python opencv-contrib-python --no-build --no-deps
uv venv -q && cargo run -q --profile fast-build pip install --no-progress --link-mode symlink opencv-python opencv-contrib-python --no-build --no-deps

We currently only catch conflicts in a single installation. Should we prime the lock database with the site-packages contents, and would that carry overhead?

TODO: Find good test packages on pypi (or make our own). The opencv-python distributions are large and the s3 package need to be built from source.

[cthoyt]

An example that's vexed me in the past:

1. https://pypi.org/project/Flask-Bootstrap/ (original, unmaintained project)
2. https://pypi.org/project/Bootstrap-Flask/ (maintained fork, uses same package name but different on PyPI)

both install to flask_bootstrap

[charliermarsh]

I don't know that we should make this user-facing, to be honest. Isn't it going to trigger any time anyone installs Jupyter?

[konstin]

I strongly think this should be user-facing, I linked some of the issues this would have prevented above (we had another one just today: #13550), especially since this is otherwise non-deterministic, silent and almost impossible to figure out if you don't know what you're looking for.

It doesn't warn for uv pip install jupyter, is there a reason we would expect it to? Note that we're checking for an __init__.py, so this does not affect namespace packages that are allowed to interleave.

[atinary-bvollmer]

Hey @charliermarsh,

I'm the author of #13550. Thanks to @konstin I found the issue but I would strongly side with here that especially for less experienced users this can lead to very frustrating problems. A warning during the install would have most likely prevented it for me.

Thanks!

[konstin]

Added a test case

[konstin]

One place where this could fail is the Python 2 path = import("pkgutil").extend_path(__path__, name) hack, but I haven't seen that one in a while, and we can special case it if necessary.

[zanieb]

(I'll review this)

[codspeed-hq[bot]]

CodSpeed WallTime Performance Report

Merging #13437 will not alter performance

_{Comparing konsti/warn-on-module-conflicts (f18ca49) with main (8968d78)}

Summary

✅ 3 untouched benchmarks

[MalteEbner]

Suggested solution: I think it should be possible for the user to define different packages under the same module name similar to the sources logic. Something similar to this:

[tool.uv.sources]
pillow = [
  { package = "pillow-simd>=9,<10", marker = "platform_machine == 'x86_64'"},
  { package = "pillow>=11,<12", marker = "platform_machine == 'aarch64'"},
]

[konstin]

Suggested solution: I think it should be possible for the user to define different packages under the same module name similar to the sources logic. Something similar to this:

[tool.uv.sources]
pillow = [
  { package = "pillow-simd>=9,<10", marker = "platform_machine == 'x86_64'"},
  { package = "pillow>=11,<12", marker = "platform_machine == 'aarch64'"},
]

~~This is already supported and recommended, since this is a valid usage it's not affected by this PR.~~

Sorry is misread the comment.

[geofft]

I sort of thought I've seen pre-PEP-420 namespace packages recently enough. Debian Code Search for path:__init__.py extend_path shows a good amount of hits, though several are in vendored and possibly old packages. Protobuf has a google/__init__.py in its sources, but it doesn't seem to actually ship it in the wheel. The latest PyQt6 wheel does have a PyQt6/__init__.py, but I'm not sure if anything else installs into that namespace.

[edit: meant to write "pre-PEP-420", not "PEP-420"]

[konstin]

Valid PEP 420 packages that don't clash are not affected by this change.