Rewrite Python interpreter discovery

[zanieb]

Updates our Python interpreter discovery to conform to the rules described in #2386, please see that issue for a full description of the behavior. Briefly, we now will search for interpreters that satisfy a requested version without stopping at the first Python executable. We also add the plumbing necessary to request Python implementations other than CPython, though we do not add support for other implementations at this time.

A major internal goal of this work is to prepare for user-facing managed toolchains i.e. fetching a requested version during uv run. These APIs are not introduced, but there is some managed toolchain handling as required for our test suite.

Some noteworthy implementation changes:

• The uv_interpreter::find_python module has been removed in favor of a uv_interpreter::discovery module.
• There are new types to help structure interpreter requests and track sources
• Executable discovery is implemented as a big lazy iterator and is a central authority for source precedence
• uv_interpreter::Error variants were split into scoped types in each module

Remaining work:

• [ ] Write and merge new test cases to main
• [x] Fix Windows test cases
• [ ] Explore behavior around implementation precedence (i.e. CPython over PyPy)
• Future: Combine PythonVersion and VersionRequest
• Future: Consider splitting ManagedToolchain into local and remote variants

Refactors split into:

• #3329
• #3330
• #3331
• #3332

Closes #2386

[codspeed-hq[bot]]

CodSpeed Performance Report

Merging #3266 will not alter performance

_{Comparing zb/interp-request-ii (1b212ab) with main (dfd6ccf)}

Summary

✅ 12 untouched benchmarks

[zanieb]

Example test setup

gh pr checkout 3266
cargo build
alias uv=$(pwd)/target/debug/uv
uv version

I'll do some local QA and share helpful commands here

[charliermarsh]

Not sure if intentional, but I have pypy (which is PyPy at Python 2.7) and pypy39 (which is PyPy at Python 3.9) in path, and cargo run venv --python pypy fails because it tries to use the one called pypy:

puffin on  zb/interp-request-ii [$] is 📦 v0.1.44 via 🐍 v3.11.8 via 🦀 v1.78.0 took 2s
❯ cargo run venv --python pypy
   Compiling uv v0.1.44 (/Users/crmarsh/workspace/puffin/crates/uv)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.75s
     Running `target/debug/uv venv --python pypy`
  × Can't use Python at `/opt/homebrew/bin/pypy`
  ╰─▶ Python executable does not support `-I` flag. Please use Python 3.8 or newer.

puffin on  zb/interp-request-ii [$] is 📦 v0.1.44 via 🐍 v3.11.8 via 🦀 v1.78.0 took 2s
❯ cargo run venv --python pypy3.9
   Compiling uv v0.1.44 (/Users/crmarsh/workspace/puffin/crates/uv)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.02s
     Running `target/debug/uv venv --python pypy3.9`
Using Python 3.9.18 interpreter at: /opt/homebrew/bin/pypy3.9
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate

Should it resolve to the pypy3.9?

cargo run venv --python pypy3.9 does work as expected.

[charliermarsh]

Otherwise, my own smoke-testing works as expected.

[charliermarsh]

I would personally expect this to work -- I copied a Python to foo locally, and --python ./foo discovers it but --python foo does not:

puffin on  zb/interp-request-ii [$?]
❯ cargo run venv --python foo
   Compiling uv v0.1.44 (/Users/crmarsh/workspace/puffin/crates/uv)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.87s
     Running `target/debug/uv venv --python foo`
  × Requested Python executable `foo` not found in PATH

puffin on  zb/interp-request-ii [$?]
❯ cargo run venv --python ./foo
   Compiling uv v0.1.44 (/Users/crmarsh/workspace/puffin/crates/uv)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.72s
     Running `target/debug/uv venv --python ./foo`
Using Python 3.9.18 interpreter at: foo
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate

I think we fixed this in the prior implementation and that it was considered a bug.

[zanieb]

I think your pypy example fails because we don't have support for detecting that as an implementation name yet (that will happen in a follow up pull request). Here, cpython is the only implementation name variant (just to reduce scope).