`pip_parse()` / `pip_repository()` hard to use is a cross-platform setup with `python_interpreter_target`

[lummax]

🚀 feature request

Relevant Rules

pip_parse() / pip_repository() or everything that exposes python_interpreter_target.

Description

pip_parse() / pip_repository() can take a python interpreter via a label with python_interpreter_target. This works fine util we run into a cross-platform setup. Consider the following setup:

• a python_linux repository extracted from a relocatable python_linux.tar.gz where we have a bin/python
• a python_windows repository extracted from a relocatable python_windows.zip where we have a python.exe

We can not use select() in the WORKSPACE.bazel, we can not use an alias() rule pointing to the correct python binary based on the execution platform (which I learned in https://github.com/bazelbuild/bazel/issues/14644#issuecomment-1022238818)).

We can't even duplicate the pip_parse() call in our WORKSPACE.bazel and create a python_deps_linux and python_deps_windows repository as we want to load() the install_deps() macro from them.

Describe the solution you'd like

Two options come to mind:

1. add an attribute python_interpreter_repository similar to python_interpreter_target where internally some OS specific suffix and a //:bin/python or //:python.exe is appended (similar to what is done in https://github.com/bazelbuild/rules_nodejs/blob/9dddd19bc9bc91cb5a0fd932e3cc2d601133a9ea/nodejs/private/node_labels.bzl#L24)
2. add an attribute python_interpreter_target_dict which maps from a OS specific identifier (https://github.com/bazelbuild/rules_nodejs/blob/9dddd19bc9bc91cb5a0fd932e3cc2d601133a9ea/nodejs/private/os_name.bzl#L32) to a label for the python interpreter

Describe alternatives you've considered

Creating a custom repository_rule() implements solution 2) by symlinking the right repository/python interpreter.

[lummax]

I did 2) as an ugly hack in https://github.com/axivion/rules_python/commit/961c0cff13897ab88e4cc714e2f9f224818acc41 which works for us. I am happy to work on cleaning it up and upstreaming it if that is an acceptable direction.

[thundergolfer]

@lummax does this basic approach shown in https://thundergolfer.com/bazel/python/2021/06/25/a-basic-python-bazel-toolchain/ not work for your use-case?

In a .bzl file you just have conditionals and grab the platform type from repository_ctx.os.name.lower().

[lummax]

In theory maybe, in our use-case unfortunately not. But apart from our use-case the approach shown in your link always requires a custom repository rule that does the platform detection instead of using http_archive(). While not impossible this is a hurdle to every adopter of this rule set?

Your example works with Linux/Macos I guess, but not if we consider Windows. On Windows the python executable is called python.exe (and exists top-level in the repository directory). On Unix the python executable is bin/python. So even if we download different archives for different platforms, we still have different executable paths to call.

Our use-case: I am in the process of migrating a code base using make where we pull in relocatable python interpreters build with conan. So I've set up an integration where we have a local_repository() pointing to a directory containing the extracted python interpreter files. The directory is always the same for every platform, but the contents differ.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"

[WillEngFlow]

Just pinging this issue as we are also interested in a solution here.

We are building on many different platforms and thus have 5 different Python interpreters. Specifically we are using the pip_parse rule to install dependencies from a requirements.txt. If we set python_interpreter_target to fix one platform we automatically break all other platforms.