feat(pex): support cross builds of py_pex_binary targetting Linux

[plobsing]

Cross builds of bundled applications would be really nice to be able to do. Often, we're bundling for the purposes of distributing an artifact, and in many cases the ultimate execution targets for the distributed artifact do not necessarily match the build machine.

The PyPex action itself can support cross-builds in a fairly straightforward way: use the cfg="exec" Python to run the PEX bundling code, rather than the cfg="target" Python. The former is what this ruleset's PEX wrapper is already running as, so all the must be done is to not request an alternative interpreter when calling into the PEX library. Unfortunately, that's precisely what the tool had been doing.

For instance, prior to this change, building an x86-64 Linux PEX binary on Macos resulted in failures with diagnostics of the form:

pex.executor.Executor.ExecutionError: OSError(8, 'Exec format error') while trying to execute `['/private/var/tmp/_bazel_peter/0576637e8cc9ee3ad4c778fd98f4cba7/external/python_toolchain_x86_64-unknown-linux-gnu/bin/python3.9', [...]

The way rules_py is using PEX, changing the interpreter should not have other meaningful effects. Perhaps if we were allowing PEX to resolve requirements freely, changing the interpreter would change the resolution results. However, that's non-hermetic, and the rules_py PEX wrapper already explicitly controls what dependencies are used and where they come from.

Unfortunately, we are only able to support cross-builds targetting Linux at the current time (and perhaps MacOS in release rulesets). Other accidental complexity blocks targetting of other platforms. See https://github.com/aspect-build/rules_py/issues/625 for details. However, even limited to just Linux as a cross target, the feature is valuable as Linux is very popular in certain contexts.

---

Changes are visible to end-users: yes

• Searched for relevant documentation and updated as needed: yes
• Breaking change (forces users to change their own code or config): no
• Suggested release notes appear below: no

Test plan

• Covered by existing test cases
• New test cases added

[aspect-workflows[bot]]

Test

1 test target passed

Targets

//py/tests/py-pex-binary:cross_builds [k8-fastbuild]                 30ms

Total test execution time was 30ms. 41 tests (97.6%) were fully cached saving 1m 10s.

[plobsing]

Test failure is a true positive. It detected a case where cross builds still don't work: cross-compiling to Mac from Linux. The reason it does this is because the Mac Rust toolchains have target_compatible_with restricting to only Mac OS. However, that's actually not what we're trying to do with these Rust toolchains; we're using Rust to compile py/tools/venv_bin which is itself a toolchain (that is, the Rust binary we're building will run on the build host, a Linux machine). I believe there's a transition missing somewhere, although I'm not sure where yet. Help definitely appreciated.

[plobsing]

Test failure is a true positive. It detected a case where cross builds still don't work: cross-compiling to Mac from Linux. The reason it does this is because the Mac Rust toolchains have target_compatible_with restricting to only Mac OS. However, that's actually not what we're trying to do with these Rust toolchains; we're using Rust to compile py/tools/venv_bin which is itself a toolchain (that is, the Rust binary we're building will run on the build host, a Linux machine). I believe there's a transition missing somewhere, although I'm not sure where yet. Help definitely appreciated.

FYI, after investigating, I've determined that the best path forward is to downscope this PR to just cross builds that target Linux. Which is still useful for building PEX bundles for things like AWS Lambda.

There is additional complexity in expanding cross-build support to targets beyond Linux. I think doing that would be neat, and https://github.com/aspect-build/rules_py/issues/625 describes in greater detail what is needed there. Support for non-Linux cross build targets still needs the changes from this PR, its just there are also other, orthogonal, additional things that must be fixed.

[CLAassistant]

All committers have signed the CLA.