meson-python icon indicating copy to clipboard operation
meson-python copied to clipboard

Support for macOS cross-architecture builds

Open dnicolodi opened this issue 3 years ago • 2 comments

Python wheels are often cross-compiled on macOS to arm64 hosts from x86_64 build machines. This is enabled by the macOS compilers allowing to cross-compile for amd64 and x86_64 with the -arch amd64 and -arch x86_64 command line arguments. setuptools invented the ARCHFLAGS environment variable to specify this arguments. The platform string returned by Python sysconfig.get_platform() can be ovrerwritten via the (undocumented) _PYTHON_HOST_PLATFORM environment variable. setuptools requires both these variables to be set correctly to generate the right object files in a wheel archive with the right tags. Other build tools like scikit-build and maturin rely only on the ARCHFLAGS environment variable. Established tools to build redistributable wheels such as cibuildwheel set both variables. However, the requirement to have to set two redundant variables in the correct way is a bit of a nuisance. (@henryiii is this a correct summary?)

The Meson way to setup a cross-build is via a cross-build-definition file https://mesonbuild.com/Cross-compilation.html#cross-compilation because it most commonly requires specifying the path (or at least the name) of the cross-compilers to be used. The situation for the macOS toolchain however, could be supported simply putting the right -arch flags in CFLAGS and similar variables.

I think there is some impedance matching here. At a minimum we need to support overwriting the wheel tag.

As an interim solution I would like to support overwriting the platform tag with the _PYTHON_HOST_PLATFORM environment variable. In combination with setting CFLAGS would allow to build arm64 wheels from x86_64 build machines.

dnicolodi avatar Nov 22 '22 12:11 dnicolodi

Scikit-build-core (and I think Maturin) only uses the ARCHFLAGS, since it would be a bug to set _PYTHON_HOST_PLATFORM and ARCHFLAGS differently. Most other tools use ARCHFLAGS to control the target architecture, but also use wheel, which is controlled by _PYTHON_HOST_PLATFORM.

It's fine to be able to specify a cross file explicitly, but the existing Python way of setting things should be supported.

henryiii avatar Nov 22 '22 13:11 henryiii

Meson delegates computing the extension filename suffix to the sysconfig module, thus there is no way around setting the _PYTHON_HOST_PLATFORM environment variable.

dnicolodi avatar Nov 23 '22 12:11 dnicolodi

Fixed in #226.

dnicolodi avatar Jan 31 '23 16:01 dnicolodi