tensorstore icon indicating copy to clipboard operation
tensorstore copied to clipboard

Add support for ppc64le architecture

Open allegro0132 opened this issue 1 year ago • 4 comments

Hello, devs,

Background

Thanks for contributing such a wonderful project to the world. Now JAX has become a platform for running machine learning and scientific computation in IBM ppc64le system(Power8, Power9). The pre-built jax and tensorflow wheels have been provided by Open-CE environment https://github.com/open-ce/open-ce/issues/549 . However, as the key of JAX ecosystem, tensorstore does not have ppc64le support yet.

Target

The PR drift working on building tensorstore in a Power system with Linux, after patching some of .bazel files. The compiling was successfully finished, and the built wheel seems to be working as expected.

Next steps

My compiling steps are shown in https://github.com/google/tensorstore/issues/146 I am not sure if the modification of Bazel is corrected or not, and if the tensorstore works as expected or not... In the next stage, I will validate the compiling by tests, and make my codes looks better.

Thanks in advance!

allegro0132 avatar Mar 20 '24 09:03 allegro0132

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

google-cla[bot] avatar Mar 20 '24 09:03 google-cla[bot]

Thanks --- overall this looks great!

The bazelisk changes seem kind of questionable and specialized --- ideally it could just work for ppc64 without specifying a base URL. Perhaps you can ask for review in the bazelisk repository (https://github.com/bazelbuild/bazelisk) --- though they may be reluctant to use a non-official build. Note that there is a go version of bazelisk in addition to the python version that we use, and the go version is not relevant to us.

Incorporating basic support like this would be easy enough but without a way to test it in our CI system it will be difficult to maintain and likely get broken soon. As we don't have ppc64 machines available for testing, and building under qemu is too slow, we will need to get cross-compilation from x86_64 set up. There are various ways that could be accomplished but I expect the easiest will be to use to https://github.com/bazel-contrib/toolchains_llvm?tab=readme-ov-file in combination with a sysroot extracted from the manylinux_2_28_ppc64le image.

jbms avatar Mar 20 '24 18:03 jbms

As far as the pypa_grpcio error when running the tests --- that is just used for a small number of tests and we could just exclude those tests on ppc64.

You can run test ... -k to see all of the failures.

jbms avatar Mar 20 '24 18:03 jbms

Please also complete the CLA so that we will be able to use your changes.

jbms avatar Mar 20 '24 18:03 jbms

Note: AOM may want to enable CONFIG_RUNTIME_CPU_DETECT for PPC, as well as HAVE_VSX

laramiel avatar May 09 '24 00:05 laramiel