Expose Python interpreter in a platform-independent way

[jiawen]

Context

I'm trying to use rules_conda with pybind11_bazel in Hermetic Python mode. This is necessary because pybind11 requires extension modules to be built for the correct Python version.

pybind11_bazel supplies a python_configure rule that takes an attr python_interpreter_target that's a of type Label:

python_configure(
  name = "local_config_python",
  python_interpreter_target = "@python_interpreter//:python_bin",
)

Problem

There does not seem to be a way to pass the Python interpreter created by rules_conda's environment to python_configure.

My configuration

In WORKSPACE:

load("@rules_conda//:defs.bzl", "conda_create", "load_conda", "register_toolchain")

load_conda(
    quiet = False,  # use True to hide conda output
    version = "4.10.3",  # optional, defaults to 4.10.3
)

conda_create(
    name = "conda_test_env",
    timeout = 600,  # each execute action can take up to 600 seconds
    clean = False,  # use True if you want to clean conda cache (less space taken, but slower subsequent builds)
    environment = "@//:conda_test_env.yml",  # label pointing to environment.yml file
    quiet = False,  # use True to hide conda output
)

register_toolchain(
    py3_env = "conda_test_env",
)

Which generates a external/conda_test_env/BUILD file containing:

py_runtime(
    name = "python_runtime",
    files = glob(
        ["conda_test_env/**/*"],
        exclude_directories = 0,
    ),
    interpreter = "conda_test_env/bin/python",
    python_version = "PY3",
)

On the file system, the Python interpreter is located at: $(bazel info output_base)/external/conda_test_env/conda_test_env/bin/python.

Things I tried:

In WORKSPACE:

python_configure(
    name = "local_config_python",
    python_interpreter_target = "@conda_test_env//conda_test_env/bin/python",
)

This doesn't work because it complains that bin is not a package: it requires a BUILD file which makes sense. Changing .../bin/python to /.../bin:python returns a similar error.

Manually adding a BUILD file to the bin directory now causes a problem for py_runtime, which contains the line: interpreter = "conda_test_env/bin/python"

ERROR: /private/var/tmp/_bazel_jiawen/8678712aa06452e8f0efd934ed354368/external/conda_test_env/BUILD:6:11: Label '@conda_test_env//:conda_test_env/bin/python' is invalid because '@conda_test_env//conda_test_env/bin' is a subpackage; perhaps you meant to put the colon here: '@conda_test_env//conda_test_env/bin:python'?

But adding the colon also fails, because apparently it must be a target name and not a label?

ERROR: /private/var/tmp/_bazel_jiawen/8678712aa06452e8f0efd934ed354368/external/conda_test_env/BUILD:6:11: @conda_test_env//:python_runtime: invalid label 'conda_test_env/bin:python' in attribute 'interpreter' in 'py_runtime' rule: invalid target name 'conda_test_env/bin:python': target names may not contain ':'

Any ideas?

[GabrielDougherty]

I don't have a definitive answer, just some guesses to try.

The colon needs to be the place in the path that has the BUILD file. Going with that, I would try @conda_test_env:conda_test_env/bin/python or maybe @conda_test_env:py_runtime if the runtime happens to be executable. Also I don't know if your program is multi platform, but on Windows the bin/ is omitted. Running the rules_conda example code on Windows, my interpreter goes to $(bazel info output_base)/external/py3_env/py3_env/python.exe.

If above doesn't work, you can try to make the python interpreter available as a filegroup:

in external/conda_test_env/BUILD:

...
filegroup(
    name = "python_interpreter",
    srcs = "conda_test_env/bin/python",
)

This could be accomplished by patching the file template in env.bzl.

If none of these work, sorry about that. If something does work, we can incorporate it into rules_conda if needed.

[yibum]

I think @GabrielDougherty is almost correct, use @conda_test_env//:conda_test_env/bin/python as python_interpreter_target would work (suppose conda_test_env is the name of environment). But I don't test the setting on Windows, something might be needed in rules_conda to make this compatible on all platforms.

Here is my local WORKSPACE

pip_install(
    requirements = "//:requirements.txt",
    name = "ext_deps",
    python_interpreter_target = "@conda_env//:conda_env/bin/python"
)

http_archive(
    name = "rules_conda",
    sha256 = "c5ad3a077bddff381790d64dd9cc1516b8133c1d695eb3eff4fed04a39dc4522",
    url = "https://github.com/spietras/rules_conda/releases/download/0.0.6/rules_conda-0.0.6.zip"
)

load("@rules_conda//:defs.bzl", "conda_create", "load_conda", "register_toolchain")

load_conda(
    version = "4.10.3",
    quiet = False,  # use True to hide conda output
)

conda_create(
    name = "conda_env",
    timeout = 600,  # each execute action can take up to 600 seconds
    clean = False,  # use True if you want to clean conda cache (less space taken, but slower subsequent builds)
    environment = "@//:conda_environment.yml",  # label pointing to environment.yml file
    quiet = False,  # use True to hide conda output
)

register_toolchain(
    py3_env = "conda_env",
)

[jiawen]

Thanks everyone! @yibum you are right! That worked!

It turns out I had a fundamental misunderstanding of how labels really work. Labels are always relative to a package root and doesn't need to be in the same directory as a package - a subdirectory (and not subpackage) is fine.

SO explains it nicely - the Bazel documentation also explains it but very tersely in the //my/app/main:testdata/input.txt example.

[spietras]

So yes, as @GabrielDougherty and @yibum already stated, if you are on Linux (or Mac, idk, I'm too poor) and your environment is named conda_env then you can use @conda_env//:conda_env/bin/python to refer to the Python interpreter. If you are on Windows this would not work, as @GabrielDougherty pointed out, because then the interpreter is at @conda_env//:conda_env/python.exe.

Making it work on all platforms would require making a symlink in the environment repository that will be called the same everywhere and point to different files on different platforms. However, symlinks are the last resort as they are not so friendly on Windows.

Or python_configure could support passing labels to runtimes, not only strings to interpreter files, then we could use @conda_env:py_runtime.

[spietras]

I will convert this issue to be more about discussing the need to expose the interpreter in a platform-independent way.

[jiawen]

Thanks! I'm glad this turned into something useful rather than just fixing my silly build issues.

python_configure does take a label - but it's to an interpreter. Maybe what we should do is, in our generated BUILD file, add a exports_files or filegroup so that the interpreter can be referred to by label rather than direct "label-to-file".

[jiawen]

Sorry in advance for the meta-discussion. @spietras Do you prefer to discuss over issues? Or GitHub "discussions"? Or perhaps a gitter.im room?

[spietras]

python_configure does take a label - but it's to an interpreter. Maybe what we should do is, in our generated BUILD file, add a exports_files or filegroup so that the interpreter can be referred to by label rather than direct "label-to-file".

exports_files only states that you can refer to that file from outside the repository, but you still need to use the full path. BUILD files in environment repositories don't have this statement and I'm surprised it worked for you without it.

filegroup would be great if it worked. However, I'm afraid you can't really use it as a path, because it's intended to describe multiple files. So if you pass a label pointing to a filegroup to rctx.path() what would you get? I suppose it will be an error, but we should check that.

There is also alias but I think it doesn't work with files, only with other targets. But again, we should double-check.

And the last possibility is symlinking. I hope it's not the only one working, but I'm afraid it is.

Sorry in advance for the meta-discussion. @spietras Do you prefer to discuss over issues? Or GitHub "discussions"? Or perhaps a gitter.im room?

If it's about something that should potentially change or be added then it's fine to discuss it in issues. If it's only a question about "how to do something" or "help, it doesn't work, but I think I am the one doing something wrong" then the Discussions tab is better for that. This one is for sure about something to change or add so we can stick here.

[GabrielDougherty]

I made a WORKSPACE (source repo here) and tried some things that did not work:

### this does not work (try #1):

filegroup(
    name = "python_interpreter",
    srcs = ["@py3_env//:py3_env/bin/python","@py3_env//:py3_env/python.exe"]
)

python_configure(
    name = "local_config_python",
    python_interpreter_target = ":python_interpreter",
)

### This does not work (try #2):
python_configure(
    name = "local_config_python",
    python_interpreter_target =  select({
        "@bazel_tools//src/conditions:windows": "@py3_env//:py3_env/python.exe",
        "//conditions:default": "@py3_env//:py3_env/bin/python",
  }),
)

### This does not work (try #3):
load("@//:interpreter.bzl", "interpreter")
interpreter("py3_env")
python_configure(
    name = "local_config_python",
    python_interpreter_target = "@://py3_env_interpreter",
)

### This works but is not platform-independent:
python_configure(
    name = "local_config_python",
    python_interpreter_target = "@py3_env//:py3_env/bin/python",
)

try #1 outputs:

 % bazel build app:number_py_ext.so
ERROR: Traceback (most recent call last):
	File "/Users/gabriel/ws/bazeltest/WORKSPACE", line 65, column 10, in <toplevel>
		filegroup(
Error in filegroup: filegroup cannot be in the WORKSPACE file (used by //external:python_interpreter)
ERROR: error loading package 'external': Package 'external' contains errors
INFO: Elapsed time: 0.382s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

try #2 outputs:

% bazel build app:number_py_ext.so
INFO: Repository local_config_python instantiated at:
  /Users/gabriel/ws/bazeltest/WORKSPACE:65:17: in <toplevel>
Repository rule python_configure defined at:
  /private/var/tmp/_bazel_gabriel/56bce0f1453b2053ba5a6eb3090e3d39/external/pybind11_bazel/python_configure.bzl:390:35: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_python':
   got value of type 'select' for attribute 'python_interpreter_target' of python_configure rule 'local_config_python'; select may not be used in repository rules
ERROR: Error fetching repository: got value of type 'select' for attribute 'python_interpreter_target' of python_configure rule 'local_config_python'; select may not be used in repository rules
ERROR: /Users/gabriel/ws/bazeltest/app/BUILD:19:17: //app:number_py_ext.so depends on @local_config_python//:python_headers in repository @local_config_python which failed to fetch. no such package '@local_config_python//': got value of type 'select' for attribute 'python_interpreter_target' of python_configure rule 'local_config_python'; select may not b
e used in repository rules
ERROR: Analysis of target '//app:number_py_ext.so' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.765s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (13 packages loaded, 50 targets configured)

Try #3 was when I also tried moving the filegroup call into a .bzl file but that didn't work because I got:

Error in glob: 'native.glob' can only be called during the loading phase
# or:
Error in filegroup: filegroup cannot be in the WORKSPACE file (used by //external:py3_env_interpreter)

---

Basically I could only use a filegroup in a BUILD file instead of WORKSPACE file. And using .bzl files didn't help. I think if the interpreter was configurable after analysis time somehow, this could work. Or another way would be if there were a bifurcation of Windows and non-Windows interpreters like:

python_configure(
    name = "local_config_python",
    python_interpreter_target = "@conda_test_env//:conda_test_env/bin/python",
    python_interpreter_target_windows = "@conda_test_env//:conda_test_env/python.exe",
)

And a bifurcation of windows/non-windows BUILD rules, then one could use select() to switch the dependent rules for any targets. That would require a significant amount of redesign/implementation time on their end (pybind11_bazel) though.