Support patch-level Python versions

[rickeylev]

The basic feature request is to allow patch-level specificity of Python versions e.g. python.toolchain(python_version="3.1.2").

Supporting this allows builds to be a bit more reproducible, as it removes the variability of different rules_python versions selecting a different patch level. That said, it's also rather niche. Patch level differences are usually inconsequential.

aignas@ also points out:

different Python toolchain versions come from different indygreg toolchain releases and some of them may have different packaging behaviour, like the last release contains the following changelog: <various updates to pip, musl, openssl, sqlite, libxzma, etc>

Making this work cleanly requires addressing a few things, though:

• If two modules (or configs) specify different patch levels, what happens?
• Toolchain repositories are named e.g. python_X_Y; these names aren't supposed to be public, but usage of them is fairly common.
• Version-pinning rules generate //X.Y:defs.bzl files.
• PyPI repo names contain the X.Y version, e.g. pip_38_foo.
• Does the packaging ecosystem support patch-level specificity? e.g. pip resolution, environment markers, whl naming formats, etc.

cc @aignas

Few potential ideas:

• Allow the root module to specify patch-level version. It has precedence. This allows projects to pin to specific versions if they want to. Submodules are not allowed to use patch-level specificity; not sure whether to silently ignore, ignore with warning, or error.

[aignas]

FYI, this is working:

diff --git a/examples/bzlmod/MODULE.bazel b/examples/bzlmod/MODULE.bazel
index 0d1c7a7..9dbdfd5 100644
--- a/examples/bzlmod/MODULE.bazel
+++ b/examples/bzlmod/MODULE.bazel
@@ -18,7 +18,7 @@ python.toolchain(
     configure_coverage_tool = True,
     # Only set when you have mulitple toolchain versions.
     is_default = True,
-    python_version = "3.9",
+    python_version = "3.9.18",
 )

 # We are also using a second version of Python in this project.
@@ -35,7 +35,7 @@ python.toolchain(
 # See the tests folder for various examples on using multiple Python versions.
 # The names "python_3_9" and "python_3_10" are autmatically created by the repo
 # rules based on the `python_version` arg values.
-use_repo(python, "python_3_10", "python_3_9", "python_versions")
+use_repo(python, "python_3_10", "python_versions", python_3_9 = "python_3_9_18")

 # This extension allows a user to create modifications to how rules_python
 # creates different wheel repositories.  Different attributes allow the user
@@ -89,7 +89,7 @@ use_repo(pip, "whl_mods_hub")
 # the Python interpreter to run to resolve dependencies.
 pip.parse(
     hub_name = "pip",
-    python_version = "3.9",
+    python_version = "3.9.18",
     requirements_lock = "//:requirements_lock_3_9.txt",
     requirements_windows = "//:requirements_windows_3_9.txt",
     # These modifications were created above and we

So it means that #1340 can be closed. The defition of done for this ticket would be to document this behaviour.

[martis42]

@aignas @rickeylev I disagree with this being a documentation issue. The recommended undocumented workaround does not work for me. One can tell python.toolchain(...) to use a specific patch version which works just fine as long as one does not use pip.parse (aka a project without external dependencies). But pip.parse does not work with the proposed workaround.

See this minimal example:
MODULE.bazel

bazel_dep(
    name = "rules_python",
    version = "0.27.0",
)

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    python_version = "3.10.11",
)

# Uncommenting this has no effect on the error
# use_repo(python, "python_versions", python_3_10 = "python_3_10_11")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "pip",
    python_version = "3.10.11",
    requirements_lock = "//:requirements.txt",  # content is irrelvant for reproducing error
)
use_repo(pip, "pip")

BUILD

load("@rules_python//python:defs.bzl", "py_binary")
load("@pip//:requirements.bzl", "requirement")

py_binary(
    name = "foo",
    srcs = ["foo.py"],
    deps = [requirement("some_dep")],  # concrete dep is irrelevant for error
)

Executing bazel run //:foo yields the error:

ERROR: Traceback (most recent call last):
        File ".../external/rules_python~0.27.0/python/private/bzlmod/pip.bzl", line 298, column 30, in _pip_impl
                _create_whl_repos(module_ctx, pip_attr, hub_whl_map, whl_overrides)
        File ".../external/rules_python~0.27.0/python/private/bzlmod/pip.bzl", line 91, column 17, in _create_whl_repos
                fail((
Error in fail: Unable to find interpreter for pip hub 'pip' for python_version=3.10.11: Make sure a corresponding `python.toolchain(python_version="3.10.11")` call exists
ERROR: error evaluating module extension pip in @@rules_python~0.27.0//python/extensions:pip.bzl

I am using rules_python 0.27.0 and Bazel 7.0.0 on Linux Mint 21.1