Configurable download of cross platform extensions

[cwaeland]

Hi,

I've begun working with rules_docker and creating py3 images. However, we have a dependency on grpcio which of course has a native extension built in. Since the python binary is built on the host machine it's bundling the Darwin version of the extension into the linux container.

I've had success passing ["--platform=manylinux2010_x86_64"] to pip_install but was wondering if this was in anyway configurable.

I stumbled across this issue https://github.com/bazelbuild/rules_python/issues/260 where you mentioned that it was possible to define multiple repos and use select to switch between them.

I took a first pass at this using the alias feature that was recently added to rules_pip:

WORKSPACE

# Linux 
pip_import(
    name = "py3_deps_manylinux2010_x86_64",
    compile = False,
    python_interpreter = "python3",
    requirements = "//3rdparty/python3:requirements.transitive.txt",
)

load(
    "@py3_deps_manylinux2010_x86_64//:requirements.bzl",
    pip3_manylinux2010_x86_64_install = "pip_install",
)

pip3_manylinux2010_x86_64_install(["--platform=manylinux2010_x86_64", "--prefer-binary"])

# Darwin
pip_import(
    name = "py3_deps",
    compile = False,
    python_interpreter = "python3",
    requirements = "//3rdparty/python3:requirements.transitive.txt",
)

load(
    "@py3_deps//:requirements.bzl",
    pip3_install = "pip_install",
)

pip3_install()

BUILD

config_setting(
    name = "alt_pip",
    define_values = {
        "python_platform": "linux",
    },
)


alias(
    name = "grpc",
    actual = select({
        "//:alt_pip": "@py3_deps_manylinux2010_x86_64//:grpcio",
        "//conditions:default": "@py3_deps//:grpcio",
    }),
)

The issue I'm running into however is that the pip install that "goes first" wins and the subsequent install already sees the packages as downloaded.

Is there any documentation you can provide to help here?

[ali5h]

we have the option to change the prefix of the names of repos generated by pip_import, you can change it to something other than pypi and then they don't overwrite each other

[cwaeland]

Ahh I see that in the recent commits. It doesn't look like it's part of the current release. I'll go ahead and test out that specific commit though.

In general though, is this approach reasonable? Or would you recommend something else?

Thanks!

[ali5h]

yes, it is recent, i think it is reasonable

[ali5h]

another solution is that we can include platform in the generated repo name

[ali5h]

basically the whole wheel signature

[cwaeland]

Yeah, I think automatically adding the whole wheel signature would be a great enhancement.