feat(toolchains): create a toolchain from a locally installed Python

[rickeylev]

[work in progress prototype]

This is a draft of a repo rule that takes a path to a python interpreter, then figures out all the details necessary to define a toolchain implementation.

It currently creates platform toolchain, i.e, it sets py_runtime.interpreter_path. This means the runtime isn't included in the runfiles.

[rickeylev]

Alright, I mentioned this in chat, but I'll post a more full description of some of the problems a local auto configuring toolchain encounters.

Ideally, we want something like this:

# The user inputs something like this:
local_runtime(
  name = "linux_runtime",
  path = "/usr/bin/python3" # Turns out to be python 3.10
)
local_runtime(
  name = "windows_runtime",
  path = "C:\Program Files\whatever\python3.exe" # turns out to be python 3.11
)

# And it generates something like this:

toolchain(
  name = "linux_toolchain",
  toolchain = "@linux_runtime//:runtime",
  target_compatible_with = [linux],
  target_settings = [is_python_3.10],
)
toolchain(
  name = "windows_toolchain",
  toolchain = "@windows_runtime//:runtime"
  target_compatible_with = [windows],
  target_settings = [is_python_3.11],
)

Unfortunately, I don't see a way to do that without triggers the evaluation of the @linux_runtime and @windows_runtime repos. This is because of the constraint settings:

In order to know the TCW values, we have to...IDK. I think manually associate it with the local_runtime() call? But, that also seems wrong -- /usr/bin/python3 is valid for both linux and mac. So I guess transform rctx.os to a constraint sting (but we want to ignore rctx.os on linux if its a windows path)? Maybe the host constraint is the other option?

For the target settings, we have to run python in order to determine the Python value. But a linux host can't run the windows path and vice versa. So now...idk. Omit target_settings? Consider the whole toolchain incompatible?

But, lets say its the happy path -- we can run the given path, get the version, figure out the os, etc. This is all computed within the runtime repo rule. If a different repo rule generates the toolchain() calls, that info needs to be passed in -- which means in order to access that info, it has to trigger evaluation of the runtime repo. This is true even if we do something like toolchain(target_compatible_with=["@linux_runtime//:os_alias"]) (where os is an alias to the appropriate @platforms target) -- even though its a target reference, toolchain evaluation will have to resolve that target, and thus trigger evaluation of the repo.

The only paths I see to deal with all this are:

1. Just accept evaluation will occur. It should be relatively cheap? If a runtime can't be setup (e.g. linux trying to run a windows path), silently fail (create like an empty no-op runtime structure or something).
2. Require (or optionally allow; requiring seems to defeat the point of an auto-configuring thing) passing in platform and Python version info. If we want to get advanced, we can write the platform and version into a separate repo if we know it at repo-evaluation time (this allows resolving the constraints, while avoiding evaluating the runtime repo itself)
3. Make the local_toolchain(...) api have platform-specific arg stuff. e.g. local_runtime(windows_path="C:\...", linux_path="...", ...); though I'm not sure how to encode python version. It seems like a full and complete set of args would be like a list of (str path, list platform_constraints, list target_settings) tuples or something.

[aignas]

I like the approach 1. out of your listed ones. It should be cheap to just have:

def _linux_impl(rctx):
    if not _is_linux(rctx):
        rctx.file("BUILD.bazel", """\
toolchain(
    name = "linux_toolchain",
    toolchain = "@linux_runtime//:runtime",
    target_compatible_with = ["@platforms//:incompatible"],
    target_settings = [],  # maybe we have to add an incompatible target setting here.
)"""
        return

    ...

Because the repository rule is not doing any network IO and the failure is really fast, then I think we should be good here.

Talking about python versions, do you want this toolchain to be used when is_python_3.x is true? What about the default case when the python_version is unset? If we have hermetic toolchains also in the mix, which one is preferred?

I think that if we could affect the default value of //python/config_settings:python_version based on the result of this linux toolchain or the python extension evaluation, it would be super nice, because that could simplify a lot of code that we have that handles the case where the python_version string flag is unset, but we have to default to something.

[rickeylev]

do you want this toolchain to be used when is_python_3.x is true?

Yes. We end up having to run python no matter what, so we have the version. Integrating it with the version-aware parts just requires adding the target_settings value. So it should be easy.

If we have hermetic toolchains also in the mix, which one is preferred?

By default, the local ones, because they're opt in. It's technically up to the user, though, since they ultimately can futz with the toolchain registration order.

[rickeylev]

OK, ready for review.

I've been sitting on this awhile, so I pulled back the scope to just the repository rules that can set up the runtime and toolchains. It works, but you can see that there's a decent amount of configuration needed in MODULE.bazel. Integrating it into the python bzlmod extension can clean that up, but it got surprisingly messy. I'll post in chat about some of the design/behavior things we need to decide on. Using it in WORKSPACE is probably similar, but I haven't tried it with workspace builds yet.

[aignas]

BTW, I realized that the current hermetic toolchain understands if it is Linux GLIBC. Is it possible to detect if the interpreter is built for MUSL vs GLIBC?

[rickeylev]

After a bit of searching, yes, i think so. The info looks to be in the platforms module: https://stackoverflow.com/questions/72272168/how-to-determine-which-libc-implementation-the-host-system-uses

[aignas]

Regarding glibc, if you look at the hermetic toolchain, we are specifying the flag value to be set to glibc (I haven't got to adding musl toolchains yet), but what it essentially is doing is including the right interpreter when building docker images, etc.

I know that this is meant for local toolchains, so is low priority IMHO, but it seems that having the flag_values in the target_settings include the glibc might mean that we can correctly handle precedence of toolchains if we are targeting the musl libc.

[rickeylev]

Ok, I was a bit wrong about how easy detecting musl is.

platform has a libc_ver() function, which can return the libc implementation, but it doesn't know how to indicate something is musl. It'll just return empty string. Which maybe that'll have to suffice.

The other option is to use packaging, but that isn't part of the stdlib, and the way it works is a decent amount of code. A small ELF format parser to find some path to something, then runs it to get some output. There's a PR for python itself, https://github.com/python/cpython/pull/103784, however, it seems to have stalled out.

Since it seems like the answer is to somehow parse something out of the executable's ELF information, I tried some various system utils to do the same, but couldn't figure out how.

Looking through sysconfig, there do appear to be several mentions of musl in some variables, e.g. as part of command line argument. So that could be a weak signal.

[aignas]

Thanks for digging.

Regarding use of packaging, it is used in all of the whl metadata parsing (and env marker eval), so using it should be OK.

We could vendor it if needed, just like what pip is doing with it's dependencies, or download it like we do for whl_library. I would probably be +1 on vendoring it since it is such a critical piece of code in the bootstrap.

On 20 July 2024 08:55:21 GMT+09:00, Richard Levasseur @.***> wrote:

Ok, I was a bit wrong about how easy detecting musl is.

platform has a libc_ver() function, which can return the libc implementation, but it doesn't know how to indicate something is musl. It'll just return empty string. Which maybe that'll have to suffice.

The other option is to use packaging, but that isn't part of the stdlib, and the way it works is a decent amount of code. A small ELF format parser to find some path to something, then runs it to get some output. There's a PR for python itself, https://github.com/python/cpython/pull/103784, however, it seems to have stalled out.

Since it seems like the answer is to somehow parse something out of the executable's ELF information, I tried some various system utils to do the same, but couldn't figure out how.

Looking through sysconfig, there do appear to be several mentions of musl in some variables, e.g. as part of command line argument. So that could be a weak signal.

-- Reply to this email directly or view it on GitHub: https://github.com/bazelbuild/rules_python/pull/2000#issuecomment-2240766073 You are receiving this because you commented.

Message ID: @.***>