bazel-hermetic-python
bazel-hermetic-python copied to clipboard
kku1993
Using with py_test
I have followed your instructions and your example does not seem to work with py_test. How do you cause the installed toolchain to be used instead of the host interpreter?
Hi @chrislovecnm, thanks got giving this a try.
I just added a py_test example to the repo and it seems to work for me. Can you share more detail on our setup?
Thanks for the fast response! I am importing your example into my project so that we can compare apples to apples.
have imported your example, into my code, and I am still getting the same error.
Here is my PR: https://github.com/cockroachdb/cockroach-operator/pull/115
Here is my error:
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //hack:verify_boilerplate_runner
-----------------------------------------------------------------------------
/usr/bin/env: 'python': No such file or directory
I am on a Debian 10 server using bazel 3.4.1. This server does not have a python binary install, it only has python3. Also, I need to do CI on a server with has, of course, python 2 installed on it.
The command that I am running is:
bazel test //hack/...
I broke up the python script into a library in order to see if that fixed the problem. I would rather just have one python file.
Here is my tool chain debug as well:
bazel build --logging=6 --toolchain_resolution_debug --incompatible_py3_is_default --python_version=PY3 //hack/...
DEBUG: Rule 'rules_python' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1592977862 -0700"
DEBUG: Repository rules_python instantiated at:
no stack (--record_rule_instantiation_callstack not enabled)
Repository rule git_repository defined at:
/home/clove/.cache/bazel/_bazel_clove/6daf43ef28bfabb61bbb29fb23d1665c/external/bazel_tools/tools/build_defs/repo/git.bzl:195:33: in <toplevel>
INFO: Build option --toolchain_resolution_debug has changed, discarding analysis cache.
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host,
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host,
INFO: ToolchainResolution: Looking for toolchain of type @bazel_tools//tools/cpp:toolchain_type...
INFO: ToolchainResolution: Looking for toolchain of type @bazel_tools//tools/python:toolchain_type...
INFO: ToolchainResolution: Considering toolchain @local_config_cc//:cc-compiler-armeabi-v7a...
INFO: ToolchainResolution: Considering toolchain @bazel_tools//tools/python:_autodetecting_py_runtime_pair...
INFO: ToolchainResolution: Toolchain constraint @platforms//cpu:cpu has value @platforms//cpu:arm, which does not match value @platforms//cpu:x86_64 from the target platform @local_config_platform//:host
INFO: ToolchainResolution: For toolchain type @bazel_tools//tools/python:toolchain_type, possible execution platforms and toolchains: {@local_config_platform//:host -> @bazel_tools//tools/python:_autodetecting_py_runtime_pair}
INFO: ToolchainResolution: Toolchain constraint @platforms//os:os has value @platforms//os:android, which does not match value @platforms//os:linux from the target platform @local_config_platform//:host
INFO: ToolchainResolution: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a, because of target platform mismatch
INFO: ToolchainResolution: Considering toolchain @local_config_cc//:cc-compiler-k8...
INFO: ToolchainResolution: For toolchain type @bazel_tools//tools/cpp:toolchain_type, possible execution platforms and toolchains: {@local_config_platform//:host -> @local_config_cc//:cc-compiler-k8}
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @local_config_cc//:cc-compiler-k8, type @bazel_tools//tools/python:toolchain_type -> toolchain @bazel_tools//tools/python:_autodetecting_py_runtime_pair
INFO: ToolchainResolution: Looking for toolchain of type @bazel_tools//tools/cpp:toolchain_type...
INFO: ToolchainResolution: Considering toolchain @local_config_cc//:cc-compiler-armeabi-v7a...
INFO: ToolchainResolution: Toolchain constraint @platforms//cpu:cpu has value @platforms//cpu:arm, which does not match value @platforms//cpu:x86_64 from the target platform @local_config_platform//:host
INFO: ToolchainResolution: Toolchain constraint @platforms//os:os has value @platforms//os:android, which does not match value @platforms//os:linux from the target platform @local_config_platform//:host
INFO: ToolchainResolution: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a, because of target platform mismatch
INFO: ToolchainResolution: Considering toolchain @local_config_cc//:cc-compiler-k8...
INFO: ToolchainResolution: For toolchain type @bazel_tools//tools/cpp:toolchain_type, possible execution platforms and toolchains: {@local_config_platform//:host -> @local_config_cc//:cc-compiler-k8}
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @local_config_cc//:cc-compiler-k8
INFO: ToolchainResolution: Looking for toolchain of type @bazel_tools//tools/jdk:runtime_toolchain_type...
INFO: ToolchainResolution: Considering toolchain @bazel_tools//tools/jdk:dummy_java_runtime...
INFO: ToolchainResolution: For toolchain type @bazel_tools//tools/jdk:runtime_toolchain_type, possible execution platforms and toolchains: {@local_config_platform//:host -> @bazel_tools//tools/jdk:dummy_java_runtime}
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/jdk:runtime_toolchain_type -> toolchain @bazel_tools//tools/jdk:dummy_java_runtime
INFO: ToolchainResolution: Looking for toolchain of type @bazel_tools//tools/jdk:toolchain_type...
INFO: ToolchainResolution: Considering toolchain @bazel_tools//tools/jdk:dummy_toolchain...
INFO: ToolchainResolution: For toolchain type @bazel_tools//tools/jdk:toolchain_type, possible execution platforms and toolchains: {@local_config_platform//:host -> @bazel_tools//tools/jdk:dummy_toolchain}
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/jdk:toolchain_type -> toolchain @bazel_tools//tools/jdk:dummy_toolchain
INFO: Analyzed 10 targets (0 packages loaded, 15190 targets configured).
INFO: Found 10 targets...
INFO: Elapsed time: 0.926s, Critical Path: 0.03s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
I'm thinking that this is a bazel version issue. I just downloaded your project and executed:
bazel build //:main
bazel test //:main_test
And I am getting the above error as well. Also, the container rules that you are using do not work with the bazel version as well :(
I'm on ubuntu 18.04 with bazel 3.1. Let met try it out in a debian container.
One thing I did notice from your PR is that you don't have register_toolchains in your WORKSPACE, see [1]
[1] https://github.com/kku1993/bazel-hermetic-python/blob/e4f3987f4efa1124b8d58738bde5130d988f181a/WORKSPACE#L112
So the toolchain is now registered. So that is fixed, but still getting the same error.
I was able to repro the issue on debian 10 with bazel 3.4.1.
I believe this is an issue with bazel - when doing bazel run or bazel test, bazel adds a wrapper script that uses #!/usr/bin/env python. See [1] and its related issue https://github.com/bazelbuild/bazel/issues/8685
One workaround for now is to install symlink python to python3 :/
[1] https://github.com/bazelbuild/bazel/blob/d95bfbded2466bb6ea796254f3f271a73505993b/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt#L1
One workaround, for now, is to install symlink python to python3 :/
In the directory that we install python in or on the system?
On the system.
On Wed, Jul 29, 2020, 18:05 Chris Love [email protected] wrote:
One workaround, for now, is to install symlink python to python3 :/
In the directory that we install python in or on the system?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kku1993/bazel-hermetic-python/issues/2#issuecomment-666013919, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASYHTTXVQTC27DIJHELWB3R6DBOZANCNFSM4PMDOTMA .
Your code itself is still run using the hermetically built python interpreter. The issue here is that bazel's wrapper script that launches the hermetically built python intepreter depends on the system's python.
I get you, and really appreciate the help!
Hermetic to me means I don’t break outside the sandbox. What is the point of this if I have to mess with the systems python. I may as well use a container. .
Since you understand this a lot better than I do, how should this be fixed? I’m pinging Simone at google to see if the priority of this bug can be raised.
So dumb question. Would this work.
If we add a symbolic link inside of the ~/.cache/bazel/_bazel_clove/6daf43ef28bfabb61bbb29fb23d1665c/external/python_interpreter/bazel_install/bin directory, and then add that folder to the path, then env would find the file.
You can find the base of that path with a bazel info output_base call.
So
bazel test --action_env="PATH=$(bazel info output_base)/external/python_interpreter:$PATH" //hack/...
sorta works, but now I am trying to figure out how to set the path more gracefully.
Since you understand this a lot better than I do, how should this be fixed? I’m pinging Simone at google to see if the priority of this bug can be raised.
Looks like there's an open PR for this already: https://github.com/bazelbuild/bazel/pull/11434. I think this is part of the fix, and the second part is to pump the python toolchain information to the script.
If we add a symbolic link inside of the
Yeah I think that works! A more graceful way to set it would be to add the --action_env flag to .bazelrc in the root of the repo. See https://docs.bazel.build/versions/master/guide.html#bazelrc-the-bazel-configuration-file