pypa
How to detect macos architecture inside setup.py
Description
I would like to change paths for include and libs in setup.py (or change symlinks during before-build), depending on whether cibuildwheel is building for x86_64 or arm64 on MacOS. Is there an environment variable or something similar that I can depend on?
I've tried using platform.version() and platform.uname(), but they always show x86_64.
Build log
No response
CI config
No response
Looks like it can be accessed by inspecting sys.executable. Is there a better way to do this?
Oh, you want this to support cross-builds. Simplest thing would be to check the _PYTHON_HOST_PLATFORM env var, and fall back to platform.machine() if it's not set. See cibuildwheel/macos.py for values.
There might be a cleaner method using the sysconfig built in module.
edit: just looked it up, I'd suggest sysconfig.get_platform() - that will do the fallbacks for you. e.g.
import sysconfig
sysconfig_platform = sysconfig.get_platform()
is_arm64 = sysconfig_platform.endswith('arm64')
is_universal2 = sysconfig_platform.endswith('universal2')
is_x86_64 = sysconfig_platform.endswith('x86_64')
Keep in mind, that tells you about the Python interpreter you are compiling with if _PYTHON_HOST_PLATFORM isn't set. Specifically, is_universal2 could be true on either x86 or ARM if you have a universal build of Python. That's fine, just remember to handle that correctly in your setup.py. :)
Yep. Though I think that detail echos what most build backends would do, too?
Here, the interest is in changing paths based on the target, so you'd need special handing for the universal case - there I think you'd have to assume you are not cross-compiling, since I assume you can't support making universal wheels when doing something like this. But you need to be able to compile and run on a universal interpreter - the official Pythons are universal, for example, so I'd assume you'd fall back to the native arch.
sysconfig.get_platform() works well for me. Thanks!
Is it something that's worth adding to the docs? If so, I would be happy to submit a PR. Otherwise, feel free to close the issue.
We could add a note to the FAQ around this area - https://cibuildwheel.readthedocs.io/en/stable/faq/#how-to-cross-compile "Detecting the target architecture when cibuildwheel is cross-compiling" - noting the _PYTHON_HOST_PLATFORM mechanism and the sysconfig.get_platform() method. If you have time that would be much appreciated @reynoldsnlp !
I'm also curious about this in non-Mac cases.
Say I'm cross-compiling for aarch64 on Linux (with an x86_64 host). In this case, I want to leave out a bunch of ext_modules that compile AVX2 libraries, since there's obviously 0% chance they would be appropriate for the target, even if there's a 99% chance the host has AVX2.
Is a call to sysconfig.get_platform going to return the target architecture, or the host builder's architecture?
We don’t support cross compiling on Linux, only windows and macOS. Linux has other issues (like being forced to use the “system” compiler which is too old to support the new architecture), so it’s generally not useful to pursue traditionally. The Zig compiler sounds like a much more likely path to try.
I think _PYTHON_HOST_PLATFORM works on any system, though.
+1 to all Henry said.
Just a little note on terminology-
Is a call to
sysconfig.get_platformgoing to return the target architecture, or the host architecture?
It'll return the 'target' i.e. the platform the wheel you're building is gonna run on. Confusingly, in compiler terminology, that's also referred to as the 'host' platform (the machine that does the compiling is called the 'build' system). I suspect that's why the environment variable is called _PYTHON_HOST_PLATFORM.