wheel Python 3.11 MacOS x86: Wheels are always marked as `universal2`

I have two seperate projects that use two different approaches to build Platform Wheels, and both generate an universal2 wheel on MacOS 3.11 with Python x86_64 that in reality is NOT an universal2 wheel, but rather x86 only. I think there is something wrong with the wheel platform detection.

The first project uses the old school python setup.py bdist_wheel approach and is a setuptools+setuptools_rust project. Example GitHub Actions run that shows universal2 being generated: https://github.com/SkyTemple/skytemple-rust/actions/runs/6355353799/job/17263321360
The second project uses pip wheel and uses setuptools+setuptools_dso: https://github.com/SkyTemple/tilequant/actions/runs/6225727358/job/16896773186

This does not happen with 3.8 - 3.11.

I am only able to reproduce this with GitHub Actions, since I don't have a Mac. I could imagine this being an issue in GitHub Actions, but I find that to be unlikely. I haven't really found any other project that publishes these kind of wheels and doesn't use GitHub Actions, so I was not able to confirm this.

Sep 30 '23 17:09 theCapypara

By default, it picks up whatever CPython is built with. If you use a universal2 copy of Python, it guesses you are making universal wheels too. If you are making redistrubtable wheels, I highly recommend cibuildwheel, which does all of the settings for you, including making sure it uses the official CPython releases, as CIs often don't provide the backward compat (10.9) that the official downloads do. But you can work out all this by hand if you really want to. I don't remember if this one is an envvar, or if you just give delocate the correct arch when you post-process the wheel (which you should already be doing if you are distributing them - cibuildwheel does this part for you too).

Oct 17 '23 12:10 henryiii

Alright, with cibuildwheel and explicitly setting the architecture this works. The only pain is the inconsistent naming of architectures between Windows and MacOS, but that's another issue...

But in general, is this behavior documented somewhere? Or what it means to "post-process" a wheel? I can't find anything on that on the packaging guides.

Oct 17 '23 14:10 theCapypara

I have the same issue exactly in my current project and I've written about it extensively in one of my project's devnotes here.

I'll post the last part which explores possible solutions:

Tried the following:

Prefix ARCHFLAGS='-arch {ARCH} to python3 setup.py
```
ARCHFLAGS='-arch x86_64' python3 setup.py bdist_wheel
```
This does force the contents of the wheel to be {ARCH} but does not change its tag which remains universal2.

Set the tag using --plat-name {tag-name} to python3 setup.py bdist_wheel:

python3 setup.py bdist_wheel --plat-name macosx_13_x86_64

This doesn't work and gives an error while testing the wheel:

ERROR: cyfaust-0.0.3-cp311-cp311-macosx_13_x86_64.whl is not a supported wheel on this platform.

What to do?

Jan 20 '24 03:01 shakfu

You are building with a universal2 build of Python. Setuptools will ask Python what it was built as and uses that. You can set _PYTHON_HOST_PLATFORM (along with ARCHFLAGS) to override this. (Though cibuildwheel does all this for you, FYI, code here).

Jan 20 '24 05:01 henryiii

@henryiii

Thanks very much for your help on this. I will check out the cibuildwheel code!

Jan 20 '24 05:01 shakfu

Thanks again, @henryiii

In case anyone else faces the same issues I did, here's some illustrative code which resolves the issue as per your advice:

#!/usr/bin/env python3

import os
import platform
import sys


def build_wheel(universal=False):
    """wheel build config (special cases macos wheels) for github runners
    
    ref: https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/macos.py
    """
    assert sys.version_info.minor >= 8, "applies to python >= 3.8"

    _platform = platform.system()
    _arch = platform.machine()
    _cmd = "python3 setup.py bdist_wheel"

    if _platform == "Darwin":
        _min_osx_ver = "10.9"

        if _arch == 'arm64' and not universal:
            _min_osx_ver = "11.0"

        if universal:
            _prefix = (f"ARCHFLAGS='-arch arm64 -arch x86_64' "
                       f"_PYTHON_HOST_PLATFORM='macosx-{_min_osx_ver}-universal2'")
        else:
            _prefix = (f"ARCHFLAGS='-arch {_arch}' "
                       f"_PYTHON_HOST_PLATFORM='macosx-{_min_osx_ver}-{_arch}'")

        _cmd = " ".join([_prefix, _cmd])

    os.system(_cmd)

if __name__ == '__main__':
    build_wheel()

Jan 20 '24 12:01 shakfu

In my case, another workaround I found was to manually override the platform tag., i.e., this code snippet

from setuptools import setup
from wheel.bdist_wheel import bdist_wheel, get_platform

class CustomWheel(bdist_wheel):
    """Override platform tags when building a wheel."""

    def initialize_options(self):
        super().initialize_options()


    def finalize_options(self):
        platform_name = get_platform("_")  # any `str` object works
        if ("universal2" in platform_name):
            self.plat_name = platform_name.replace("universal2", "x86_64")  # or your host/target architecture


    def run(self):
        super.run()


setup(cmdclass={"bdist_wheel": CustomWheel}, *args, **kwargs)

works for me, though in hindsight I should have used the tuple that is returned from bdist_wheel.get_tag instead. Providing --plat-name through user options for my command class does not seem to work unless I switch to the deprecated python setup.py bdist_wheel comand, which is not recommended of course.

I did end up switching to cibuildwheel, though, since it manages to do this as discussed in the previous comments on this thread.

Jan 20 '24 17:01 agriyakhetarpal

Thanks for the alternative solution, @agriyakhetarpal

I actually tried the --plat-name way previously and couldn't get it to work..

Ideally, in the case that a universal2 python building universal2 wheels by default, there should be a flag such as python setup.py bdist_wheel --native to force the wheel to be built in the native architecture.

Jan 20 '24 17:01 shakfu

I'd highly recommend using cibuildwheel. There are several other concerns; for example, you don't want to use the GitHub Actions compiled Python or brew's compiled Python, as those don't target macOS 10.9, like the official binaries do, so you should always download the official binaries and use those when building redistributable wheels. Cross-compiling to Windows ARM is even more complex, and targeting manylinux/musllinux requires a different workflow. cibuildwheel solves all of these issues, and can be used locally too if you want. It also works even if you are not using wheel, such as when using maturin (Rust), scikit-build-core, or meson-python, none of which can use wheel since there's no public API (and maturin is also itself written in Rust). pip and build are designed to simply build wheels for your current system, while cibuildwheel is designed to build redistributable wheels (using pip or build internally). There are a few other frameworks for this if you don't like cibuildwheel, like multibuild, though as far as I know those are tied to CI, while cibuildwheel (ironically) is a program you can run anywhere.

Flags should not be added to the python setup.py interface, that has been deprecated for years. PEP 517 is the preferred interface.

I'd also avoid customizing bdist_wheel if possible, it's been stated that the programmatic interface is not intended to be public.

Jan 21 '24 02:01 henryiii

Thanks for the advice, @henryiii

I'd highly recommend using cibuildwheel. There are several other concerns; for example, you don't want to use the GitHub Actions compiled Python or brew's compiled Python, as those don't target macOS 10.9, like the official binaries do, so you should always download the official binaries and use those when building redistributable wheels. Cross-compiling to Windows ARM is even more complex, and targeting manylinux/musllinux requires a different workflow. cibuildwheel solves all of these issues, and can be used locally too if you want. It also works even if you are not using wheel, such as when using maturin (Rust), scikit-build-core, or meson-python, none of which can use wheel since there's no public API (and maturin is also itself written in Rust). pip and build are designed to simply build wheels for your current system, while cibuildwheel is designed to build redistributable wheels (using pip or build internally). There are a few other frameworks for this if you don't like cibuildwheel, like multibuild, though as far as I know those are tied to CI, while cibuildwheel (ironically) is a program you can run anywhere.

In my particular case, for the current project I'm working on, the build process is a little bit involved already with a setup.py file with two setup() functions to accommodate a default dynamically linked build, and an alternative statically linked variant. Since the project is essentially a cython wrapping of part of a largish c++ dsp framework, prior to running the normal setup.py/cython extension build process, I have to, via a fit-for-purpose python script, download, build and install the c++ framework into a local prefix...

Everything was working fine on local machines, until I decided to add github actions to the equation to help build across a number of platforms {macos, linux} x {arm64, x86_64} and then I had the issues which you previously helped me to resolve.

During the latter part of the above process, I actually looked at cibuildwheel, but I thought I might as well get the simpler case working before I took a deep dive into cibuildwheel, which seemed as per the name and the way it is presented a solution tailored for ci builds.

Flags should not be added to the python setup.py interface, that has been deprecated for years. PEP 517 is the preferred interface.

Understood.

I'd also avoid customizing bdist_wheel if possible, it's been stated that the programmatic interface is not intended to be public.

Fair enough.

Jan 21 '24 06:01 shakfu

I'd highly recommend using cibuildwheel. There are several other concerns; for example, you don't want to use the GitHub Actions compiled Python or brew's compiled Python, as those don't target macOS 10.9, like the official binaries do, so you should always download the official binaries and use those when building redistributable wheels. Cross-compiling to Windows ARM is even more complex, and targeting manylinux/musllinux requires a different workflow. cibuildwheel solves all of these issues, and can be used locally too if you want. It also works even if you are not using wheel, such as when using maturin (Rust), scikit-build-core, or meson-python, none of which can use wheel since there's no public API (and maturin is also itself written in Rust). pip and build are designed to simply build wheels for your current system, while cibuildwheel is designed to build redistributable wheels (using pip or build internally). There are a few other frameworks for this if you don't like cibuildwheel, like multibuild, though as far as I know those are tied to CI, while cibuildwheel (ironically) is a program you can run anywhere.

Flags should not be added to the python setup.py interface, that has been deprecated for years. PEP 517 is the preferred interface.

I'd also avoid customizing bdist_wheel if possible, it's been stated that the programmatic interface is not intended to be public.

Many thanks for your comments @henryiii.

If you have a project that is still using setup.py, and running bdist_wheel, and you want to move to cibuildwheel, what would be the proper way of doing it?

Invoke cibuildwheel instead of bdist_wheel from setup.py?, or
Abandon setup.py altoghether? And, in this latter case, what would be the replacement? A pyproject.toml plus cibuildwheel?

May 30 '24 15:05 rturrado

I would recommend all projects have a pyproject.toml, regardless of if they have a setup.py, it's not a replacement, even if it can be in some cases. It's a file that tells the builder what is needed to build. setup.py can't do that, since it's in Python.

Cibuildwheel is a frontend; it doesn't care if you use setup.py or something else. It triggers the build with an environment specifically designed for building redistributable wheels. So you'd run cibuildwheel and it will use pip (or build) with the right settings for building your wheels. Internally, if you use setuptools, bdist_wheel is still invoked, but since the environment is set up correctly (universal official installer is installed on macOS, SETUPTOOLS_EXT_SUFFIX is set to tell setuptools to make the correct extension suffix, etc.), it will work without you needing to customize bdist_wheel.

May 30 '24 16:05 henryiii

See https://cibuildwheel.pypa.io/en/stable/setup/ and https://learn.scientific-python.org/development/guides/gha-wheels/ for some info.

May 30 '24 16:05 henryiii