Wheel repair issue for arm64 builds on MacOS

[fwilliams]

Description

Hi,

I maintain point-cloud-utils, a python package mostly written in C++ and whose only external dependency is NumPy (on top of the usual libc++/libstdc++ of course).

I've recently updated to the latest stable cibuildwheel (v2.20.0) and my macos builds fail for arm64 targets. In particular the Repairing wheel... stage fails with the following error (link to a run here):

  Fixing: /private/var/folders/4d/0gnh84wj53j7wyk695q0tc_80000gn/T/cibw-run-n_nai2ss/cp38-macosx_arm64/built_wheel/point_cloud_utils-0.30.4-cp38-cp38-macosx_11_0_arm64.whl
  Traceback (most recent call last):
    File "/private/var/folders/4d/0gnh84wj53j7wyk695q0tc_80000gn/T/cibw-run-n_nai2ss/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/delocate/delocating.py", line 789, in _calculate_minimum_wheel_name
      arch_version[arch] = max(version, version_dkt[arch])
  KeyError: 'arm64'
  The above exception was the direct cause of the following exception:
  Traceback (most recent call last):
    File "/private/var/folders/4d/0gnh84wj53j7wyk695q0tc_80000gn/T/cibw-run-n_nai2ss/cp38-macosx_arm64/build/venv/bin/delocate-wheel", line 8, in <module>
      sys.exit(main())
    File "/private/var/folders/4d/0gnh84wj53j7wyk695q0tc_80000gn/T/cibw-run-n_nai2ss/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/delocate/cmd/delocate_wheel.py", line 110, in main
      copied = delocate_wheel(
    File "/private/var/folders/4d/0gnh84wj53j7wyk695q0tc_80000gn/T/cibw-run-n_nai2ss/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/delocate/delocating.py", line 1004, in delocate_wheel
      out_wheel_fixed = _check_and_update_wheel_name(
    File "/private/var/folders/4d/0gnh84wj53j7wyk695q0tc_80000gn/T/cibw-run-n_nai2ss/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/delocate/delocating.py", line 831, in _check_and_update_wheel_name
      new_name, problematic_files = _calculate_minimum_wheel_name(
    File "/private/var/folders/4d/0gnh84wj53j7wyk695q0tc_80000gn/T/cibw-run-n_nai2ss/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/delocate/delocating.py", line 798, in _calculate_minimum_wheel_name
      raise DelocationError(
  delocate.libsana.DelocationError: Failed to find any binary with the required architecture: 'arm64'

I updated the repair command to list the shared libraries that need to be copied over by delocated-wheel to:

CIBW_REPAIR_WHEEL_COMMAND_MACOS: delocate-listdeps --all {wheel} && delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}

which produces:

delocate-listdeps --all /private/var/folders/4d/0gnh84wj53j7wyk695q0tc_80000gn/T/cibw-run-n_nai2ss/cp38-macosx_arm64/built_wheel/point_cloud_utils-0.30.4-cp38-cp38-macosx_11_0_arm64.whl
  /usr/lib/libSystem.B.dylib
  /usr/lib/libc++.1.dylib

None of this seems suspicious to me, and I'm not quite sure why delocate-wheel is failing. Any help.

Build log

https://github.com/fwilliams/point-cloud-utils/actions/runs/10595440706/job/29361222607

CI config

name: Build and upload to PyPI

# Build on every branch push, tag push, and pull request change:
on: [push, pull_request]
# Alternatively, to publish when a (published) GitHub Release is created, use the following:
# on:
#   push:
#   pull_request:
#   release:
#     types:
#       - published

jobs:
  build_wheels:
    name: Build wheels on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-20.04, windows-2019, macos-13, macos-14]

    steps:
      - uses: actions/checkout@v3

      - name: Set up QEMU
        if: runner.os == 'Linux'
        uses: docker/setup-qemu-action@v3
        with:
          platforms: all

      - name: Build wheels
        env:
          # Skip 32-bit builds, MUSL libc, and pypy
          CIBW_SKIP: "*-win32 *-manylinux_i686 pp* *-musllinux*"
          CIBW_BEFORE_BUILD_LINUX: yum -y install gcc-gfortran lapack-devel blas-devel
          CIBW_ARCHS_LINUX: "auto"
          CIBW_REPAIR_WHEEL_COMMAND_MACOS: delocate-listdeps --all {wheel} && delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}
          # CIBW_ARCHS_MACOS: "x86_64 universal2 arm64"
        uses: pypa/[email protected]
      - uses: actions/upload-artifact@v3
        with:
          path: ./wheelhouse/*.whl

  build_sdist:
    name: Build source distribution
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Build sdist
        run: pipx run build --sdist

      - uses: actions/upload-artifact@v2
        with:
          path: dist/*.tar.gz

  upload_pypi:
    needs: [build_wheels, build_sdist]
    runs-on: ubuntu-latest
    # upload to PyPI on every tag starting with 'v'
    if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/v')
    # alternatively, to publish when a GitHub Release is created, use the following rule:
    # if: github.event_name == 'release' && github.event.action == 'published'
    steps:
      - uses: actions/download-artifact@v2
        with:
          name: artifact
          path: dist

      - uses: pypa/[email protected]
        with:
          user: __token__
          password: ${{ secrets.PYPI_API_TOKEN }}
          # To test: repository_url: https://test.pypi.org/legacy/

[Czaki]

This error code means that there is no binary blob recognized as arm64 one. It may mean that from some reason your build backend build the x86 binaries.

Easiest way to validate it will be disable the repair command to inspect produces package. I see that you use custom cmake wrapping in setup.py that may lead to such a problems.

I think that switch to scikit-build-core as build backend may solve the problem.

[fwilliams]

This seems to be a pybind11 issue.

Point cloud utils is built as a python module that imports from a _pcu_internal.so extension module. For some reason, pybind11 names the extension module _pcu_internal..cpython-3XX-darwin.so regardless of whether it's built for arm64 or x86_64.

Since point-cloud-utils has no system dependencies other than libstd++, I set CIBW_REPAIR_WHEEL_COMMAND_MACOS to "" (i.e. skip the repair process). This appears to fix the issue and the library now builds and runs on both arm64 and x86_64 macs.

[henryiii]

This looks suspicious:

https://github.com/fwilliams/point-cloud-utils/blob/9c7c54b7c1ac426d0a78854214901ec5fbaae684/setup.py#L72-L73

This is more likely to be correct:

https://github.com/pybind/cmake_example/blob/7a94877f581a14de4de1a096fb053a55fc2a66bf/setup.py#L99-L103

You don't need to try to force anything for a native compile, and your code wouldn't handle cross-compilation.

Or, much better yet, just use scikit-build-core, which is the official Python builder for CMake. See https://github.com/pybind/scikit_build_example for example. It handles all of these sort of stuff for you, and doesn't break periodically with new setuptools releases.

[Czaki]

Point cloud utils is built as a python module that imports from a _pcu_internal.so extension module. For some reason, pybind11 names the extension module _pcu_internal..cpython-3XX-darwin.so regardless of whether it's built for arm64 or x86_64.

I have downloaded artifact and checked CPU type. It looks like there is some bug in delocate (In my code). I will try to debug this and when I have done, then I will ping you.

[Parskatt]

@Czaki @fwilliams any update on this? I see @fwilliams is still using CIBW_REPAIR_WHEEL_COMMAND_MACOS: ""