Third party dependencies are incorrect when using RBE because host != exec

[groodt]

Context

This is a tracking issue to recognise the lack of support for Remote Execution (RBE) when using the support for third party dependencies from PyPI (or similar indexes/mirrors).

A reproduction of the issue has been developed in this rules_python_rbe_exec_exploration repository.

The issue occurs when there are native dependencies in the transitive closure of third party dependencies, which is very common in the Python ecosystem.

The root cause of the issue is that the current implementation that imports third party dependencies as targets of the bazel build assumes that the host platform is the same as the execution platform (host == exec). However, a very common configuration of remote execution would be macos or windows host platforms executing remote builds on linux platforms (host != exec).

Known workarounds

1. Only use pure python third party dependencies (this is not really practical)
2. Try the experimental support for multi-platform wheel repositories using experimental_index_url (follow https://github.com/bazelbuild/rules_python/issues/260 for when this becomes default)
3. Use alternative rules such as rules_pycross that can import third party dependencies where host != exec

Related

• #260
• #2949

[aignas]

I have realized that this will be (partially?) fixed by #2949 because then the dependency tree will be correctly resolved irrespective of the host platform, but the issue about pulling the right platform specific wheels will be still there until #260 is resolved.