ARM builds?

[takluyver]

This is a very speculative issue. I'm aware that PEP 513 says manylinux1 is only defined for i686 and x86_64. But it would be nice if instructions telling people to pip install compiled modules worked easily and quickly on things like the Raspberry Pi too. It might also be valuable to the few brave souls trying to make Python work on mobile platforms.

So this issue is a place to work out what would be needed.

1. Architecture requirements: there are a lot of different versions of ARM, and there are confusingly similar looking version schemes for the architectures (e.g. ARMv7) and cores (e.g. ARM11). There may also be optional modules. Debian supports three buckets of ARM architectures: armhf requires ARMv7 and a floating point unit (hf = hardware float). armel supports older architectures, and cores without an FPU. arm64 is for the newer 64-bit processors. In terms of raspis:

• Raspi 0 and 1 have ARMv6 architecture, so Debian armhf does not support them. They do have FPUs, though, so Raspian is compiled to make use of this.
• Raspi 2 has ARMv7
• Raspi 3 has a 64-bit ARMv8 architecture.

2. Minimum library versions: These probably don't need to be all that old, because the systems I think this would target are likely to be running a relatively recent distro. But we'd need to work out what they are.

3. Distro for building: What distro would be the equivalent of the Centos 5.11 we use for building x86 and x64 manylinux wheels?

4. Build platform: Are there any CI services which offer ARM builds? Is virtualisation fast enough? Can we make it easy for people to build wheels on their own Raspberry Pi or similar device.

[rmcgibbo]

I am not personally very knowledgeable about ARM, so I don't know the direct answers to any of those questions. It would be really excellent if manylinux could support ARM at some point in the future though -- I have the impression that it's a growing platform.

For the manylinux1 spec, @njsmith and I leveraged a lot of Continuum's work to define the base distro and minimum library versions. Since the manylinux1 release, Continuum has added some support for ARM to conda and their packages, so perhaps this would be a good place to begin to identify best practices.

[takluyver]

Thanks, that's good to know. It looks like they're targeting ARMv7 and using Raspbian to run the builds.

[njsmith]

Unless someone wants to do lots of abi compatibility testing across different arm distributions, I think the most viable way forward here would be to push forward with the idea of distro-specific wheel tags, so that people could upload e.g "raspbian wheels"?

On Dec 12, 2016 3:40 AM, "Thomas Kluyver" [email protected] wrote:

Thanks, that's good to know. It looks like they're targeting ARMv7 and using Raspbian to run the builds.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pypa/manylinux/issues/84#issuecomment-266407585, or mute the thread https://github.com/notifications/unsubscribe-auth/AAlOaLss6R_-ibhkVU8Q45O-oTuK5bUhks5rHTKugaJpZM4LJ_pp .

[KellenSunderland]

Using dockcross works quite well for ARM builds. I've created a few packages that use cross-compilation rather than virtualization. Builds that take hours on a Pi typically take a minute or two. This Dockerfile for example builds packages that run well for me with a Pi3 + Raspbian jessi. https://github.com/dockcross/dockcross/blob/master/linux-armv7/Dockerfile

[takluyver]

Since my motivation for this was getting Python packages on a Raspberry Pi, I'll drop a link to piwheels, @bennuttall's project to compile wheels for that platform.

[bennuttall]

An overview:

Although the hardware spec of Pi 1/Zero is ARMv6, Pi 2 is ARMv7 and Pi 3 is ARMv8, the reality is that Raspbian runs in ARMv6 user space regardless on your hardware. When you build a wheel on a Pi it's tagged armv6l on Pi 1/Zero and armv7l on Pi 2/3, but really it's always ARMv6.

So my project piwheels provides wheels for Raspberry Pi, but I wouldn't bank on the platform wheels working on other ARM boards. There's no particular need for maintainers to upload their wheels to PyPI as Pi users can use piwheels as their repo instead. ~Soon~, in Raspbian, pip will be configured to use piwheels. (update: this is now true)

[QJKX]

Python on Termux (an android Linux environment) would also benefit from arm wheels.

[NotAFile]

We're having this problem with testing ARM in CI. The lack of wheels forces us to compile all dependencies for a fresh build, which takes around 20 minutes.

[vielmetti]

There is a CI system available, Drone Cloud, that can build natively for 32-bit and 64-bit Arm systems. There's also bare metal resources in the @worksonarm project to tackle this. Happy to help tackle this, especially for arm64 (aarch64).

[vielmetti]

@bennuttall - does piwheels support 32-bit Arm only, or do you also have 64-bit arm64 support?

[bennuttall]

We only provide wheels for 32-bit systems.

[vielmetti]

Referencing https://marcin.juszkiewicz.com.pl/2019/02/23/we-need-arm64-systems-for-developers-again/ which describes the resulting pain that pip install scipy, which takes a few seconds on x86, takes hours on arm64 because of lack of wheels.

[hrw]

I discussed that with Pypi devs some time ago on irc.

Whole service depends on whatever authors upload. So if we want non-x86 wheels then we need to convince authors to build and upload arm wheels.

[hrw]

And when it comes to what to use as a minimal...

I think that Debian jessie is sensible minimum. Eventually CentOS 7.

Older releases did not supported arm64.

[ralphie-boy]

I need to do python development that can include the python SpeechRecognition package. Samsung recently released the Galaxy Book 2 with the latest 64 bit Qualcomm Snapdragon processor (850). It comes with Windows 10 S which can be upgraded to Windows Pro. Windows Pro will support WSL (Windows Subsystem for Linux). WSL will allow me seamlessly run Ubuntu. Ubuntu will have built-in Python. The question is what happens when I try to install the SpeechRecognition package? I know for example this will not work on iPad. Will the Arm 64 architecture support this package?

[takluyver]

SpeechRecognition itself appears to be a pure Python package, so there should be no problem installing it anywhere Python works. But it looks like it's a wrapper around various local and remote speech recognition engines, so the actual difficulty will probably depend which backend you need. But it's almost certainly possible - you just might need to install some things through apt or compile them from source.

[ralphie-boy]

Thanks for the reply, takluyver. I think you are correct. But I failed to procure the Galaxy Book 2 in time for my trip. Instead I'm taking my iPad, on which, Pythonista is the only supported Python. So SpeechRecognition won't work, because Pythonista is not a true complete working Python. It's pretty good, but it is limited to a subset of packages for which dependencies can be supported by whatever wrappers.

[ulope]

On a slightly related note docker has supported multi-platform images for a while now through manifest lists and all official (i.e. made by docker) images on docker hub are multi platform.

Together with the new buildx multi platform support this could make providing ARM images much easer.

[takluyver]

Just to remind everyone who might contribute here: the essential first step would be to define appropriate ARM architecture tags. What instruction sets & extensions are relevant, what are the rules for compatibility (e.g. presumably a binary built with NEON extensions won't work on a CPU that doesn't have them), and how can we detect what the hardware supports?

If we can work that out, I'd be optimistic that we could do something for glibc-based distros on ARM relatively easily, based on what's already been done for x86 & x86_64.

An alternative approach would be to focus on defining more specific tags, e.g. for raspbian. But that's obviously not as broadly useful.

[hrw]

@takluyver for ARMv7 it should follow 'armhf' defined by Debian/Fedora/etc distros. No NEON, -vfp3d16.

AArch64 (aka arm64) is simpler - just gcc defaults.

And target Debian/jessie or Debian/stretch as minimal OS version and binaries should (imho) work under CentOS 7/8, Debian 8+, Ubuntu 16.04+ etc.

[lpsinger]

It would be more convenient if the arm manylinux image was based on the same OS as the x86_64 image. Some times I need to install additional build tools and it would create some friction if I had to use yum for x86_64 but apt for arm. Is there a CentOS port for ARM?

[lpsinger]

Also I take it that an armhf does not include NEON but aarch64 does?

[mattip]

Correct me if I'm wrong. NEON and friends are not part of the spec, just like AVX is not part of the x64 spec. It is up to c-extension library writers to deal with detecting CPU capabilities and dealing with their presence or absence at run-time. NumPy does this by having separate loops for different architecture variants, as does Intel in their various packages.

[lpsinger]

Correct me if I'm wrong. NEON and friends are not part of the spec, just like AVX is not part of the x64 spec.

This is unclear to me. According to https://en.wikipedia.org/wiki/ARM_architecture#ARMv8-A:

To both AArch32 and AArch64, ARMv8-A makes VFPv3/v4 and advanced SIMD (NEON) standard.

[hrw]

It would be more convenient if the arm manylinux image was based on the same OS as the x86_64 image. Some times I need to install additional build tools and it would create some friction if I had to use yum for x86_64 but apt for arm. Is there a CentOS port for ARM?

CentOS 7 supports aarch64 as one of altarch architetures. 32bit arm port is available but with other level of support iirc.

[mayeut]

To sum-up a few comments:

Also I take it that an armhf does not include NEON but aarch64 does?

Yes

Correct me if I'm wrong. NEON and friends are not part of the spec, just like AVX is not part of the x64 spec.

This is unclear to me. According to https://en.wikipedia.org/wiki/ARM_architecture#ARMv8-A: To both AArch32 and AArch64, ARMv8-A makes VFPv3/v4 and advanced SIMD (NEON) standard.

To extend @hrw https://github.com/pypa/manylinux/issues/84#issuecomment-531679045:

ARMv7 it should follow 'armhf' defined by Debian/Fedora/etc distros. No NEON, -vfp3d16.

armhf: 32bits, minimum target armv7, No NEON, -vfp3d16

AArch64 (aka arm64) is simpler - just gcc defaults.

aarch64: 64bits, minimum target armv8, NEON by default

[lpsinger]

It would be more convenient if the arm manylinux image was based on the same OS as the x86_64 image. Some times I need to install additional build tools and it would create some friction if I had to use yum for x86_64 but apt for arm. Is there a CentOS port for ARM?

CentOS 7 supports aarch64 as one of altarch architetures. 32bit arm port is available but with other level of support iirc.

How relevant is 32-bit armhf support these days? If you only supported aarch64, would that be a problem?

If armhf is important, is the CentOS 7 support for it good enough? If not, is the Fedora picture any better?

[mattip]

I would vote for starting with 64 bit, and leaving armhf out for now.

[bennuttall]

How relevant is 32-bit armhf support these days? If you only supported aarch64, would that be a problem?

The Raspberry Pi generally runs a 32-bit operating system. Pi 3 and 4 are 64-bit CPUs but the Raspbian image runs in 32-bit mode to support all boards. Plus Pi 1 and Pi Zero are still sold, and they are armv6.

[lpsinger]

Looks like the CentOS docker images support armhf and aarch64 as of CentOS 7 and later (https://hub.docker.com/r/arm32v7/centos/, https://hub.docker.com/r/arm64v8/centos/), so ARM support based upon them would be possible as of manylinux2014 (but not manylinux2010).