rules_python
rules_python copied to clipboard
bazelbuild
"Cross compilation" of py_binary/py_image/py_library targets
Hi,
I have a py_binary that depends on a python pip library (grpcio) that has a native extension bundled in. This means that to create a linux container i'd need to have the pip_import rule download the manylinux wheel, not the host one (macos in my case).
Is there a way to force this? Otherwise py_image will happily just bundle up wheels with darwin native libs. py_binary will also only make host-runnable things.
use https://github.com/ali5h/rules_pip/ and use pip_install(["--platform=linux_x86_64"])
@ali5h this works (thanks!) but is there integration into the bazel platform selection functions? I don't want separate targets for Linux and Mac.
E.g. Something that works with https://docs.bazel.build/versions/master/platforms.html
you can define multiple piplib repos for different platforms and use select to pickup correct one
Is that documented and supported anywhere? That would be the ideal case, for bazel to automatically pick up the right pip repo for the right target platform.
E: should it not just be built in to the py_binary/py_library rule, to select the right target platform libs automatically?
This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Seems https://github.com/bazelbuild/rules_python/pull/510 and https://github.com/bazelbuild/rules_python/pull/531 and addressing the same issue? I'd love to have that functionality so excited to so folks interested enough to open PRs 😄
Hopefully @alexeagle and @meastham can combine forces to get the awesome functionality over the line! 🚀
Friendly ping to @alexeagle and @meastham, are either of you able to take another look at your pull requests? I'd love to have that functionality 😃
I wonder if pipenv could be used to facilitate generating cross platform dependency graphs. Does anyone have experience with the tool?
Friendly ping to @alexeagle and @meastham, are either of you able to take another look at your pull requests? I'd love to have that functionality 😃
Sorry for the long delay on this! I did some investigation on some of the open questions, I'll see if we can find a path to getting this merged.
I don't think the PRs are exactly addressing the same issue. #531 allows having different requirements files for different platforms, but doesn't not allow downloading wheels for a different platform than the host platform. #510 allows downloading wheels for arbitrary platforms ("cross compiling"), but requires every platform to share one requirements file which has potential problems. Conceptually they could be compatible but the implementations would need to be modified to harmonize the way platform selection is done.
I'm all for the wheel-only approach taken in #510. I don't want native compilation happening at all in the loading phase, as it can lead to hard-to-diagnose cache misses between machines (in my experience).
Thinking out loud here. If adding a wheel-only option, can we remove pip from the build-time process entirely? Instead of using pip download, why not just use bazel's built-in http tools to download the wheel? Instead of using pip-compile to generate a locked requirements.txt, write something that generates a .bzl file with compiled dependencies in a more bazel-centric fashion. Maybe resolvers from poetry or pipenv are used for this, which apparently support multi-platform resolve. Dependencies between libraries - including platform specific - could be explicit in the generated file and defined using bazel's own select or whatever.
I'm sure there are gotchas here (and significant work), but maybe detaching from pip could open up new avenues.
I think it should also be possible to compile python/C++ sources into wheels, however those need to happen in actions so they are debuggable and so that the target platform can be used.
Ping on this @alexeagle and @meastham. #510 is exactly what I need today and having #531 would be a strong nice to have. Any progress here? Thanks!
Ping on this @alexeagle and @meastham. #510 is exactly what I need today and having #531 would be a strong nice to have. Any progress here? Thanks!
Hey @gopher-maker,
I'm just blocked on getting some guidance from a repository owner on how to resolve the outstanding issues with #510 (not a complaint btw, I'm sure everybody is quite busy!).
FWIW we've been using it for a fairly large Python codebase without significant problems for about 9 months now, so if you're feeling adventurous you can use it already. It looks like it now needs some non-trivial rebasing work; I'll see if I can get to that this week.
Any leads on this yet? I am running into a similar issue trying to use Mujoco in a bazel workspace.
@f0rmiga has been working on something related to this at a client, I don't have any update sorry.
There's a current effort that @philsc is writing in a doc, and has the collaboration from @jvolkman. I wrote a resolver to download Python packages using http_file. It's similar to how Gazelle does it for Go third-party deps. I'll start a draft PR in the coming days to have it maintained in rules_python. It will change quite a bit the current workflow to work with wheels in Bazel, and I don't have all the answers yet, so I don't foresee it being in a release soon.
Ah, thanks for the answers folks! I will try to find some other workaround.
@f0rmiga Could you please post the link to the PR when it's open in this thread? I'm very much interested in this also!
I am looking to fix this for bzlmod in #1643 at least for the whl downloading part, I'll leave this ticket to track the sdist cross-compilation.
