tensorstore
tensorstore copied to clipboard
Add support for ppc64le architecture
Hello, devs,
Background
Thanks for contributing such a wonderful project to the world.
Now JAX has become a platform for running machine learning and scientific computation in IBM ppc64le system(Power8, Power9). The pre-built jax and tensorflow wheels have been provided by Open-CE environment https://github.com/open-ce/open-ce/issues/549 . However, as the key of JAX ecosystem, tensorstore does not have ppc64le support yet.
Target
The PR drift working on building tensorstore in a Power system with Linux, after patching some of .bazel files. The compiling was successfully finished, and the built wheel seems to be working as expected.
Next steps
My compiling steps are shown in https://github.com/google/tensorstore/issues/146
I am not sure if the modification of Bazel is corrected or not, and if the tensorstore works as expected or not...
In the next stage, I will validate the compiling by tests, and make my codes looks better.
Thanks in advance!
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
View this failed invocation of the CLA check for more information.
For the most up to date status, view the checks section at the bottom of the pull request.
Thanks --- overall this looks great!
The bazelisk changes seem kind of questionable and specialized --- ideally it could just work for ppc64 without specifying a base URL. Perhaps you can ask for review in the bazelisk repository (https://github.com/bazelbuild/bazelisk) --- though they may be reluctant to use a non-official build. Note that there is a go version of bazelisk in addition to the python version that we use, and the go version is not relevant to us.
Incorporating basic support like this would be easy enough but without a way to test it in our CI system it will be difficult to maintain and likely get broken soon. As we don't have ppc64 machines available for testing, and building under qemu is too slow, we will need to get cross-compilation from x86_64 set up. There are various ways that could be accomplished but I expect the easiest will be to use to https://github.com/bazel-contrib/toolchains_llvm?tab=readme-ov-file in combination with a sysroot extracted from the manylinux_2_28_ppc64le image.
As far as the pypa_grpcio error when running the tests --- that is just used for a small number of tests and we could just exclude those tests on ppc64.
You can run test ... -k to see all of the failures.
Please also complete the CLA so that we will be able to use your changes.
Note: AOM may want to enable CONFIG_RUNTIME_CPU_DETECT for PPC, as well as HAVE_VSX