rules_python icon indicating copy to clipboard operation
rules_python copied to clipboard
verified publisher icon bazelbuild

Python toolchain cache invalidation

Open ewhauser opened this issue 1 year ago • 4 comments

🐞 bug report

Affected Rule

rules_python 0.35.0

Is this a regression?

Yes

Description

The latest version of rules_python is not caching the Python toolchain correctly on OSX.

🔬 Minimal Reproduction

 load("@rules_python//python:repositories.bzl", "py_repositories", "python_register_multi_toolchains")

 py_repositories()

 default_python_version = "3.11"

 python_register_multi_toolchains(
     name = "python",
     default_version = default_python_version,
     ignore_root_user_error = True,
     python_versions = [
         "3.9",
         "3.11",
     ],
 )

 load("@python_3_11//:defs.bzl", "interpreter")
 load("@python_3_9//:defs.bzl", _interpreter_39 = "interpreter")

 # Create a central repo that knows about the dependencies needed for
 # requirements.txt.
 load("@rules_python//python:pip.bzl", "pip_parse")

 pip_parse(
     name = "pypi",
     python_interpreter_target = interpreter,
     requirements_darwin = "//:requirements.macos.compiled",
     requirements_lock = "//:requirements.compiled",
 )

🌍 Your Environment

Operating System:

  
OSX Sonoma 14.7

Output of bazel version:

  
Bazelisk version: 1.22.0
Aspect CLI version: 5.10.14
Build label: 7.3.2
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Tue Oct 1 17:48:21 2024 (1727804901)
Build timestamp: 1727804901
Build timestamp as int: 1727804901

Rules_python version:

  
0.35.0

Anything else relevant?

This occurs in both WORKSPACE and bzlmod builds.

ewhauser avatar Oct 09 '24 22:10 ewhauser

Can you provide more info? In particular, reproduction steps or output that indicates something isn't being cached.

rickeylev avatar Oct 09 '24 23:10 rickeylev

Yes, the hermetic interpreters are downloaded again and all the pip dependencies are reinstalled

On Wed, Oct 9, 2024 at 5:31 PM Richard Levasseur @.***> wrote:

Can you provide more info? In particular, reproduction steps or output that indicates something isn't being cached.

— Reply to this email directly, view it on GitHub https://github.com/bazelbuild/rules_python/issues/2283#issuecomment-2403617542, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABACPPIXUJ45BBMQV7CYJTZ2W4F3AVCNFSM6AAAAABPVPI5P6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBTGYYTONJUGI . You are receiving this because you authored the thread.Message ID: @.***>

ewhauser avatar Oct 10 '24 00:10 ewhauser

Does #2256 fix the cache invalidation you are seing?

aignas avatar Oct 10 '24 07:10 aignas

It appears so - I have not been able to reproduce using that commit. Closing.

ewhauser avatar Oct 11 '24 03:10 ewhauser

@ewhauser, could you add context for reopening this issue?

aignas avatar Oct 21 '24 01:10 aignas

Think I closed this prematurely. This behavior still seems to exist in 0.37.0. Trying to track down a min-reproduction.

ewhauser avatar Oct 21 '24 01:10 ewhauser

I bisected and this is the offending commit: 68c30483f80a55870eb113e17ab21ff65d5b97ed

ewhauser avatar Oct 25 '24 00:10 ewhauser

Interesting https://github.com/bazelbuild/rules_python/commit/68c30483f80a55870eb113e17ab21ff65d5b97ed would have been my last guess here. What is the toolchain matching log if you pass --toolchain_resolution_debug=python?

FYI: @rickeylev

aignas avatar Oct 25 '24 10:10 aignas

Yes, I agree. I've bisected it 3 separate times and always ended up at this commit. Pinned to the previous commit in our monorepo and the problem is solved.

Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
WARNING: Build option --toolchain_resolution_debug has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
Analyzing: target //tools/pre-commit:pre-commit (0 packages loaded, 0 targets configured)
Analyzing: target //tools/pre-commit:pre-commit (0 packages loaded, 0 targets configured)
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, 
INFO: ToolchainResolution: Performing resolution of @@rules_python~//python:exec_tools_toolchain_type for target platform @@platforms//host:host
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_ppc64le-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_s390x-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_ppc64le-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_s390x-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_ppc64le-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_s390x-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_ppc64le-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_s390x-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_ppc64le-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_s390x-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_4_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_4_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_ppc64le-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_4_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_s390x-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution: No @@rules_python~//python:exec_tools_toolchain_type toolchain found for target platform @@platforms//host:host.
INFO: ToolchainResolution: Performing resolution of @@bazel_tools//tools/python:toolchain_type for target platform @@platforms//host:host
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_aarch64-apple-darwin//:python_runtimes; mismatching config settings: _0000_python_3_11_aarch64-apple-darwin_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_aarch64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0000_python_3_11_aarch64-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0000_python_3_11_armv7-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0000_python_3_11_i386-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0000_python_3_11_ppc64le-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0000_python_3_11_riscv64-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_s390x-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0000_python_3_11_s390x-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-apple-darwin//:python_runtimes; mismatching config settings: _0000_python_3_11_x86_64-apple-darwin_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-pc-windows-msvc//:python_runtimes; mismatching config settings: _0000_python_3_11_x86_64-pc-windows-msvc_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0000_python_3_11_x86_64-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_aarch64-apple-darwin//:python_runtimes; mismatching config settings: _0001_python_3_8_aarch64-apple-darwin_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_aarch64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0001_python_3_8_aarch64-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0001_python_3_8_armv7-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0001_python_3_8_i386-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_ppc64le-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0001_python_3_8_ppc64le-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0001_python_3_8_riscv64-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_s390x-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0001_python_3_8_s390x-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-apple-darwin//:python_runtimes; mismatching config settings: _0001_python_3_8_x86_64-apple-darwin_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-pc-windows-msvc//:python_runtimes; mismatching config settings: _0001_python_3_8_x86_64-pc-windows-msvc_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0001_python_3_8_x86_64-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_aarch64-apple-darwin//:python_runtimes; mismatching config settings: _0002_python_3_9_aarch64-apple-darwin_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_aarch64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0002_python_3_9_aarch64-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0002_python_3_9_armv7-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0002_python_3_9_i386-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0002_python_3_9_ppc64le-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0002_python_3_9_riscv64-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_s390x-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0002_python_3_9_s390x-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-apple-darwin//:python_runtimes; mismatching config settings: _0002_python_3_9_x86_64-apple-darwin_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-pc-windows-msvc//:python_runtimes; mismatching config settings: _0002_python_3_9_x86_64-pc-windows-msvc_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0002_python_3_9_x86_64-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_aarch64-apple-darwin//:python_runtimes; mismatching config settings: _0003_python_3_10_aarch64-apple-darwin_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_aarch64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0003_python_3_10_aarch64-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0003_python_3_10_armv7-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0003_python_3_10_i386-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0003_python_3_10_ppc64le-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0003_python_3_10_riscv64-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_s390x-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0003_python_3_10_s390x-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-apple-darwin//:python_runtimes; mismatching config settings: _0003_python_3_10_x86_64-apple-darwin_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-pc-windows-msvc//:python_runtimes; mismatching config settings: _0003_python_3_10_x86_64-pc-windows-msvc_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0003_python_3_10_x86_64-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_aarch64-apple-darwin//:python_runtimes; mismatching config settings: _0004_python_3_12_aarch64-apple-darwin_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_aarch64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0004_python_3_12_aarch64-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0004_python_3_12_armv7-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0004_python_3_12_i386-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0004_python_3_12_ppc64le-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0004_python_3_12_riscv64-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_s390x-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0004_python_3_12_s390x-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-apple-darwin//:python_runtimes; mismatching config settings: _0004_python_3_12_x86_64-apple-darwin_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-pc-windows-msvc//:python_runtimes; mismatching config settings: _0004_python_3_12_x86_64-pc-windows-msvc_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0004_python_3_12_x86_64-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Toolchain @@rules_python~~python~python_3_11_4_aarch64-apple-darwin//:python_runtimes is compatible with target plaform, searching for execution platforms:
      ToolchainResolution:     Compatible execution platform @@platforms//host:host
      ToolchainResolution:   All execution platforms have been assigned a @@bazel_tools//tools/python:toolchain_type toolchain, stopping
      ToolchainResolution: Recap of selected @@bazel_tools//tools/python:toolchain_type toolchains for target platform @@platforms//host:host:
      ToolchainResolution:   Selected @@rules_python~~python~python_3_11_4_aarch64-apple-darwin//:python_runtimes to run on execution platform @@platforms//host:host
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, 
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@bazel_tools//tools/python:toolchain_type -> toolchain @@rules_python~~python~python_3_11_4_aarch64-apple-darwin//:python_runtimes
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, 
INFO: ToolchainResolution: Performing resolution of @@rules_python~//python:exec_tools_toolchain_type for target platform @@platforms//host:host
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_ppc64le-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_s390x-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_ppc64le-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_s390x-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_ppc64le-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_s390x-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_ppc64le-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_s390x-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_ppc64le-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_s390x-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_aarch64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_aarch64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_4_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_4_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_ppc64le-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_4_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_s390x-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_x86_64-apple-darwin//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_x86_64-pc-windows-msvc//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_4_x86_64-unknown-linux-gnu//:py_exec_tools_toolchain; mismatching config settings: is_exec_tools_toolchain_enabled
      ToolchainResolution: No @@rules_python~//python:exec_tools_toolchain_type toolchain found for target platform @@platforms//host:host.
INFO: ToolchainResolution: Performing resolution of @@bazel_tools//tools/python:toolchain_type for target platform @@platforms//host:host
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_aarch64-apple-darwin//:python_runtimes; mismatching config settings: _0000_python_3_11_aarch64-apple-darwin_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_aarch64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0000_python_3_11_aarch64-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0000_python_3_11_armv7-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0000_python_3_11_i386-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0000_python_3_11_ppc64le-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_11_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0000_python_3_11_riscv64-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_s390x-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0000_python_3_11_s390x-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-apple-darwin//:python_runtimes; mismatching config settings: _0000_python_3_11_x86_64-apple-darwin_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-pc-windows-msvc//:python_runtimes; mismatching config settings: _0000_python_3_11_x86_64-pc-windows-msvc_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_11_x86_64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0000_python_3_11_x86_64-unknown-linux-gnu_3.11.9_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_aarch64-apple-darwin//:python_runtimes; mismatching config settings: _0001_python_3_8_aarch64-apple-darwin_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_aarch64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0001_python_3_8_aarch64-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0001_python_3_8_armv7-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0001_python_3_8_i386-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_ppc64le-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0001_python_3_8_ppc64le-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0001_python_3_8_riscv64-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_8_s390x-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0001_python_3_8_s390x-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-apple-darwin//:python_runtimes; mismatching config settings: _0001_python_3_8_x86_64-apple-darwin_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-pc-windows-msvc//:python_runtimes; mismatching config settings: _0001_python_3_8_x86_64-pc-windows-msvc_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_8_x86_64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0001_python_3_8_x86_64-unknown-linux-gnu_3.8.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_aarch64-apple-darwin//:python_runtimes; mismatching config settings: _0002_python_3_9_aarch64-apple-darwin_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_aarch64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0002_python_3_9_aarch64-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0002_python_3_9_armv7-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0002_python_3_9_i386-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0002_python_3_9_ppc64le-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_9_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0002_python_3_9_riscv64-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_s390x-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0002_python_3_9_s390x-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-apple-darwin//:python_runtimes; mismatching config settings: _0002_python_3_9_x86_64-apple-darwin_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-pc-windows-msvc//:python_runtimes; mismatching config settings: _0002_python_3_9_x86_64-pc-windows-msvc_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_9_x86_64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0002_python_3_9_x86_64-unknown-linux-gnu_3.9.19_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_aarch64-apple-darwin//:python_runtimes; mismatching config settings: _0003_python_3_10_aarch64-apple-darwin_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_aarch64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0003_python_3_10_aarch64-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0003_python_3_10_armv7-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0003_python_3_10_i386-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0003_python_3_10_ppc64le-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_10_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0003_python_3_10_riscv64-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_s390x-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0003_python_3_10_s390x-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-apple-darwin//:python_runtimes; mismatching config settings: _0003_python_3_10_x86_64-apple-darwin_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-pc-windows-msvc//:python_runtimes; mismatching config settings: _0003_python_3_10_x86_64-pc-windows-msvc_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_10_x86_64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0003_python_3_10_x86_64-unknown-linux-gnu_3.10.14_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_aarch64-apple-darwin//:python_runtimes; mismatching config settings: _0004_python_3_12_aarch64-apple-darwin_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_aarch64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0004_python_3_12_aarch64-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_armv7-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0004_python_3_12_armv7-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_i386-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0004_python_3_12_i386-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0004_python_3_12_ppc64le-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@[unknown repo 'python_3_12_riscv64-unknown-linux-gnu' requested from @@rules_python~~python~pythons_hub]//:python_runtimes; mismatching config settings: _0004_python_3_12_riscv64-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_s390x-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0004_python_3_12_s390x-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-apple-darwin//:python_runtimes; mismatching config settings: _0004_python_3_12_x86_64-apple-darwin_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-pc-windows-msvc//:python_runtimes; mismatching config settings: _0004_python_3_12_x86_64-pc-windows-msvc_3.12.3_1
      ToolchainResolution:   Rejected toolchain @@rules_python~~python~python_3_12_x86_64-unknown-linux-gnu//:python_runtimes; mismatching config settings: _0004_python_3_12_x86_64-unknown-linux-gnu_3.12.3_1
      ToolchainResolution:   Toolchain @@rules_python~~python~python_3_11_4_aarch64-apple-darwin//:python_runtimes is compatible with target plaform, searching for execution platforms:
      ToolchainResolution:     Compatible execution platform @@platforms//host:host
      ToolchainResolution:   All execution platforms have been assigned a @@bazel_tools//tools/python:toolchain_type toolchain, stopping
      ToolchainResolution: Recap of selected @@bazel_tools//tools/python:toolchain_type toolchains for target platform @@platforms//host:host:
      ToolchainResolution:   Selected @@rules_python~~python~python_3_11_4_aarch64-apple-darwin//:python_runtimes to run on execution platform @@platforms//host:host
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@bazel_tools//tools/python:toolchain_type -> toolchain @@rules_python~~python~python_3_11_4_aarch64-apple-darwin//:python_runtimes
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, 
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, 
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, 
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, 
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, 
INFO: Analyzed target //tools/pre-commit:pre-commit (0 packages loaded, 22922 targets configured).
INFO: Found 1 target...
Target //tools/pre-commit:pre-commit up-to-date:
  /private/var/tmp/_bazel_ewhauser/c1fee43d9f70be2c9377b78115996411/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/tools/pre-commit/pre-commit
INFO: Elapsed time: 0.398s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

The other piece of information I can give is that the repro seems to be:

  1. Run bazel build //target
  2. Run bazel run //target

and the dependent repositories are invalidated. I've tried to extract this into a simple repo without luck yet.

ewhauser avatar Oct 25 '24 17:10 ewhauser

That is a weird commit to be the cause. All that local toolchain stuff isn't active.

Maybe the changes to repo_utils.bzl? It introduces some getenv() and watch() calls. Maybe something about that is over-triggering?

rickeylev avatar Oct 25 '24 18:10 rickeylev

Looking through the code, the new watch/getenv calls I see are:

  • which() calls getenv(PATH).
  • which() calls watch(binary).

which() is called as part some of the runtime repo setup, iirc.

In repo_utils (https://github.com/bazelbuild/rules_python/blob/main/python/private/repo_utils.bzl), if you:

  • Comment out L267, replace with path = 'junk'
  • Comment out L270

Does the invalidation go away?

rickeylev avatar Oct 25 '24 18:10 rickeylev

@ewhauser, is the latest RC still having the issue?

aignas avatar Dec 03 '24 08:12 aignas

Tested this morning against the RC and have not been able to reproduce.

ewhauser avatar Dec 03 '24 16:12 ewhauser

I have removed #2406 dependency from fetching of the toolchains and I wonder if that fixed it. I'll close this as done, but feel free to reopen if you see any issues.

Thanks for the testing!

aignas avatar Dec 04 '24 00:12 aignas

I've kept my eye on this bug for months, with my repository's rules_python version pinned at 0.32.2, hoping it would be resolved. With the recent release of bazel 8, it's becoming harder to keep rules_python pinned at that version, so I'd like to discuss this now.

I have a minimal repro here: https://github.com/Ubehebe/rules_python_2283_mvce. It's a trivial repo with a git pre-commit hook that runs bazel build .... What I observe is that directly building the repo, then doing the same indirectly via the pre-commit hook, causes pip dependencies to be re-collected. (Directly doing bazel build ... multiple times in a row, or invoking the pre-commit hook multiple times in a row, is idempotent as expected.) I do not observe this at rules_python 0.32.2, but do observe it at every later release, including the 1.0.0 release that others in this issue have said fixes their problems.

git has some sneaky semantics when it runs hooks. Specifically, git exposes the value of git --exec-path as a $GIT_EXEC_PATH env var for the hook. Could this env var be causing the cache invalidation? The hook doesn't invalidate pip dependencies when run directly (./pre-commit.sh), only when run indirectly via a git commit.

Ubehebe avatar Dec 22 '24 04:12 Ubehebe

I should also mention that I tried to patch rules_python 1.0.0, via bazel's single_version_override, to revert https://github.com/bazelbuild/rules_python/commit/68c30483f80a55870eb113e17ab21ff65d5b97ed, which has been identified as a culprit earlier in this discussion. The results were inconclusive. I still observed the pip dependency invalidation. However, 68c30483f80a55870eb113e17ab21ff65d5b97ed doesn't revert cleanly against 1.0.0, so it's possible I incorrectly resolved some conflicts.

Ubehebe avatar Dec 22 '24 04:12 Ubehebe

@Ubehebe, I think what you are describing could be a separate issue even if it originatesdue to similar reasons. Could you please create a new ticket?

One way to debug this could be to print bazel info instead of bazel build so that we could rule out simple discrepancies between the envs (e.g. the repository cache location).

aignas avatar Jan 01 '25 09:01 aignas