Tool `delocate-wheel` alters tag in wheel name to seemingly invalid tag/number

[andreas-el]

Describe the bug Tool delocate-wheel alters tag in wheel name to seemingly invalid tag/number We are using GitHub-runners to generate som large macOS wheels for repository Carolina

To Reproduce

The wheel before we use delocate:

  Created wheel for carolina: filename=carolina-1.0.16.dev3+g42cf9bc-cp38-cp38-macosx_11_0_x86_64.whl size=32980 sha256=f719c488eded81a29c5685e636b6b290f516d8fab9cbfde7b626041e49240904
...

Running delocate-wheel:

delocate-wheel -w /tmp/carolina_dist -v $unfixed_wheel_path
...
...
-rw-r--r--  1 runner  wheel    17M Apr  5 13:42 carolina-1.0.16.dev3+g42cf9bc-cp38-cp38-macosx_13_6_x86_64.whl

I see that there are dependencies that would indicate this to be set to 13_x, but as far as I can tell the wheel won't work when 13_6 is used. If I rename the file I can install it locally. The same is true for 11_0.

Compatible tags: 1835
  cp38-cp38-macosx_13_0_x86_64
  cp38-cp38-macosx_13_0_intel
  cp38-cp38-macosx_13_0_fat64
  cp38-cp38-macosx_13_0_fat32
  cp38-cp38-macosx_13_0_universal2
  cp38-cp38-macosx_13_0_universal
  ...

Wheels used/generated

# generated using version 0.10.7 -- works as intended
carolina-1.0.16.dev2+g9c153fd-cp311-cp311-macosx_11_0_x86_64.whl

# generated using version 0.11.0 -- works when renaming to `13_0` (or `11_0`)
carolina-1.0.16.dev3+g42cf9bc-cp311-cp311-macosx_13_6_x86_64.whl

carolina-1.0.16.dev3+g42cf9bc-cp311-cp311-macosx_13_6_x86_64.whl.tar.gz

carolina-1.0.16.dev2+g9c153fd-cp311-cp311-macosx_11_0_x86_64.whl.tar.gz

Platform (please complete the following information):

• OS versions macOS 13 and macOS 14
• Delocate version: 0.11.0

Additional context Works with previous release, 0.10.7

[HexDecimal]

Can you try setting the environment variable MACOSX_DEPLOYMENT_TARGET=11.0 (set it to whichever version you're trying to target) before running delocate-wheel?

[HexDecimal]

Which dependencies are forcing 13.6? If those are built locally then maybe set MACOSX_DEPLOYMENT_TARGET=11.0 for the entire workflow.

[andreas-el]

Can you try setting the environment variable MACOSX_DEPLOYMENT_TARGET=11.0 (set it to whichever version you're trying to target) before running delocate-wheel?

Thanks for the quick replies.

We have already set this variable, but in the case of macos-14-large this does not work when using delocate 0.11.0.

Fixing: /tmp/carolina_dist_unfixed/carolina-1.0.16.dev3+gfb7b791-cp312-cp312-macosx_11_0_x86_64.whl
  File "/Users/runner/hostedtoolcache/Python/3.12.2/x64/bin/delocate-wheel", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/delocate/cmd/delocate_wheel.py", line 110, in main
    copied = delocate_wheel(
             ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/delocate/delocating.py", line 1004, in delocate_wheel
    out_wheel_fixed = _check_and_update_wheel_name(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/delocate/delocating.py", line 839, in _check_and_update_wheel_name
    raise DelocationError(
delocate.libsana.DelocationError: Library dependencies do not satisfy target MacOS version 11.0:
/private/var/folders/02/qm7jv9vx7_ld2jykcqjpgkgw0000gn/T/tmp_kontn8r/wheel/carolina.dylibs/libboost_serialization.dylib has a minimum target of 14.0
/private/var/folders/02/qm7jv9vx7_ld2jykcqjpgkgw0000gn/T/tmp_kontn8r/wheel/carolina.dylibs/libsoga.dylib has a minimum target of 14.0
/private/var/folders/02/qm7jv9vx7_ld2jykcqjpgkgw0000gn/T/tmp_kontn8r/wheel/carolina.dylibs/libdream.dylib has a minimum target of 14.0
/private/var/folders/02/qm7jv9vx7_ld2jykcqjpgkgw0000gn/T/tmp_kontn8r/wheel/carolina.dylibs/libjega.dylib has a minimum target of 14.0
/private/var/folders/02/qm7jv9vx7_ld2jykcqjpgkgw0000gn/T/tmp_kontn8r/wheel/carolina.dylibs/libteuchoscomm.13.0.dylib has a minimum target of 14.0
/private/var/folders/02/qm7jv9vx7_ld2jykcqjpgkgw0000gn/T/tmp_kontn8r/wheel/carolina.dylibs/libconmin.dylib has a minimum target of 14.0
...

https://github.com/equinor/Carolina/actions/runs/8611588703/job/23599070801?pr=105

This has me a bit stumped though (why does the error indicate it needs target 14.0 ?)

# When exporting MACOSX_DEPLOYMENT_TARGET=11.0 for macos-14-large (x86_64)
/private/var/folders/02/qm7jv9vx7_ld2jykcqjpgkgw0000gn/T/tmp_kontn8r/wheel/carolina.dylibs/libteuchoscomm.13.0.dylib has a minimum target of 14.0

This does not affect macOS-14 arm64-builds where we had to specify 13.0 as target.

# When exporting MACOSX_DEPLOYMENT_TARGET=13.0 for macos-14 (arm64)
INFO:delocate.delocating:Modifying install name in carolina.dylibs/libteuchoscomm.13.0.dylib from @rpath/libteuchosparameterlist.13.dylib to @loader_path/libteuchosparameterlist.13.0.dylib

---

Using the following setup works fine:

• delocate 0.10.7
• MACOSX_DEPLOYMENT_TARGET=11.0
• macos-14-large

...
INFO:delocate.delocating:Modifying install name in carolina.dylibs/libteuchoscomm.13.0.dylib from @rpath/libteuchoscore.13.dylib to @loader_path/libteuchoscore.13.0.dylib
...
Output new carolina wheel to /tmp/carolina_dist
total 35704
-rw-r--r--  1 runner  wheel    17M Apr  9 07:20 carolina-1.0.16.dev4+g72314b7-cp312-cp312-macosx_11_0_x86_64.whl

Please note that I am using the same build-files when assembling this wheel. Meaning that both boost and Dakota were pre-built in during another step in the workflow. I have yet to test exporting the MACOSX_DEPLOYMENT_TARGET variable when performing the actual build.

[HexDecimal]

I'll clarify that Delocate 0.11.0 has PR #198 applied which is the cause of the "error". For other devs, this was able to reveal compatibility issues unknown before the release of Delocate 0.11.0. Delocate 0.10.7 does not verify MacOS version compatibility, Delocate 0.11.0 does and will error on any detected compatibility issues.

It's possible that the wheels being fixed by Delocate 0.10.7 are not compatible with MacOS 11.5 and are mislabeled (because Delocate 0.10.7 does not verify MacOS versions). It's possible libraries on a macos-14-large runner are not compatible with earlier versions of MacOS.

A libraries minimum MacOS version is determined from that libraries LC_BUILD_VERSION.

I'm not experienced enough to explain why the libraries from macos-14-large have strange minimum versions for its libraries compared to macos-14.

[andreas-el]

I'll clarify that Delocate 0.11.0 has PR #198 applied which is the cause of the "error". For other devs, this was able to reveal compatibility issues unknown before the release of Delocate 0.11.0. Delocate 0.10.7 does not verify MacOS version compatibility, Delocate 0.11.0 does and will error on any detected compatibility issues.

It's possible that the wheels being fixed by Delocate 0.10.7 are not compatible with MacOS 11.5 and are mislabeled (because Delocate 0.10.7 does not verify MacOS versions). It's possible libraries on a macos-14-large runner are not compatible with earlier versions of MacOS.

A libraries minimum MacOS version is determined from that libraries LC_BUILD_VERSION.

I'm not experienced enough to explain why the libraries from macos-14-large have strange minimum versions for its libraries compared to macos-14.

I just wanted to stress that the list provided by the tool regarding compatible tags does not include the one it ends up inserting into the filename. I've assumed this is a list of compatible tags for the wheel in question.

Compatible tags: 1835
  cp38-cp38-macosx_13_0_x86_64
  cp38-cp38-macosx_13_0_intel
  cp38-cp38-macosx_13_0_fat64
  cp38-cp38-macosx_13_0_fat32
  cp38-cp38-macosx_13_0_universal2
  cp38-cp38-macosx_13_0_universal
...

[HexDecimal]

Which tool/command gives a list of compatible tags?

[andreas-el]

I realise i mistook the origin of that output, wrongfully thinking it came from delocate-listdeps, but it origin is actually python -m build --wheel . --outdir /tmp/carolina_dist_unfixed.

Sorry about that.

I still don't understand why the wheels actually work when I rename them. Most tags I've seen used for wheels are on the format of 12_0, 13_0, 14_0 just indicating the major version.

[HexDecimal]

If the wheels fixed by Decloate in macos-14-large output with a later version of MacOS but still work on earlier versions then the libraries provided by that runner have an incorrect target version (LC_BUILD_VERSION is wrong and should be lower). That's my conclusion when I follow the logic at the moment.

The libraries in macos-14-large should have the same targets as macos-14 but don't. I think this because in the CI run you've posted the macos-14 builds manage to pass and this is my only explanation for why.

[jschueller]

same issue here, my wheel gets renamed with an unsupported tag: macosx_13_6_x86_64

[HexDecimal]

same issue here, my wheel gets renamed with an unsupported tag: macosx_13_6_x86_64

What was the tag you expected? And did you set MACOSX_DEPLOYMENT_TARGET?

[jschueller]

I was expecting macosx_13_0_x86_64

Now if I set MACOSX_DEPLOYMENT_TARGET=13.0 I see the reason ; it flags all my python extensions modules as incompatible (but not my c++ library on which they depend):

delocate.libsana.DelocationError: Library dependencies do not satisfy target MacOS version 13.0:
memoryview.cpython-310-darwin.so has a minimum target of 13.6
...
_statistics.cpython-310-darwin.so has a minimum target of 13.6

note that it does not happen on arm64

[HexDecimal]

I was expecting macosx_13_0_x86_64

Now if I set MACOSX_DEPLOYMENT_TARGET=13.0 I see the reason ; it flags all my python extensions modules as incompatible (but not my c++ library on which they depend):

Was that Python installation built locally? Perhaps with MACOSX_DEPLOYMENT_TARGET=13.6 in the environment?

[jschueller]

no, I'm using Python from homebrew but its not even linked to Python explicitely:

otool -L openturns/memoryview.cpython-38-darwin.so
openturns/memoryview.cpython-38-darwin.so:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.0.0)

[HexDecimal]

How does one affect the MACOSX_DEPLOYMENT_TARGET or MacOS.version of a program built by brew? I'm not experienced with it and I don't have the platform to mess around with it.

[jschueller]

It turns out it was cmake that decided set the min osx version, I worked around it with:

-DCMAKE_OSX_DEPLOYMENT_TARGET=13.0

https://cmake.org/cmake/help/latest/variable/CMAKE_OSX_DEPLOYMENT_TARGET.html

[jvolkman]

My understanding is that for any version of macos > 10, pip and packaging will only accept macosx_X_0 wheels, where X is the platform major version. So a wheel named carolina-1.0.16-cp311-cp311-macosx_13_6_x86_64.whl will always be rejected, and it should instead be named carolina-1.0.16-cp311-cp311-macosx_13_0_x86_64.whl.

See this comment and subsequent replies for discussion.

[peter-urban]

I also have this issue for the macos images in github actions (compiling c++ with clang and meson, creating module with pybind11), from what I can see:

I use the macos-12 (actually macos 12.7, and here python 3.12 but 3.11 or 3.10 give the same results) (https://github.com/themachinethatgoesping/themachinethatgoesping/actions/runs/9794288645/job/27043980546)

• pip wheel produces 'themachinethatgoesping-0.23.2-cp312-cp312-macosx_12_0_x86_64.whl'
• delocate renames this to "themachinethatgoesping-0.23.2-cp312-cp312-macosx_12_7_x86_64.whl"
• this cannot be installed because macosx_12_7 is not recognized as valid platform tag by pip
• pip -v debug outputs e.g. macosx_12_0 but no macosx_12_7

if I set MACOSX_DEPLOYMENT_TARGET=12.0 I get another error: (https://github.com/themachinethatgoesping/themachinethatgoesping/actions/runs/9807611968/job/27081707264)

delocate.libsana.DelocationError: Library dependencies do not satisfy target MacOS version 12.0:
/private/var/folders/0f/b0mzpg5d31z074x3z5lzkdxc0000gn/T/tmpawgr6nx8/wheel/themachinethatgoesping/.dylibs/libc++.1.0.dylib has a minimum target of 12.7
/private/var/folders/0f/b0mzpg5d31z074x3z5lzkdxc0000gn/T/tmpawgr6nx8/wheel/themachinethatgoesping/.dylibs/libc++abi.1.0.dylib has a minimum target of 12.7
/private/var/folders/0f/b0mzpg5d31z074x3z5lzkdxc0000gn/T/tmpawgr6nx8/wheel/themachinethatgoesping/.dylibs/libunwind.1.0.dylib has a minimum target of 12.7
/private/var/folders/0f/b0mzpg5d31z074x3z5lzkdxc0000gn/T/tmpawgr6nx8/wheel/themachinethatgoesping/.dylibs/libomp.dylib has a minimum target of 12.7

So delocate gets the minimum version from the libc++ dylibs. The question is now, is this a bug or a feature of delocate? Or to put in multiple questions:

• Is it true that compiling with macos-12.7 creates wheels that cannot run in macos-12.0 because of an incompatible libc++.1.0.dylib ?
• If so: this would be a pip bug, since in this case pip should support macos_12_7 tags?
• And also if so: does that mean we would need a kind of manylinux approach to build macos wheels? E.g. build the wheel on an actual macos12.0 docker image to make sure it works on all >=macos-12 installations?

[HexDecimal]

Is it true that compiling with macos-12.7 creates wheels that cannot run in macos-12.0 because of an incompatible libc++.1.0.dylib ?

That's true by definition. The minor version increment indicates that features may be added which would be missing in previous versions.

If so: this would be a pip bug, since in this case pip should support macos_12_7 tags?

Since macOS 11, the minor version is treated differently than previous macOS releases.

As I reread https://github.com/pypa/packaging/issues/578, it seems the upstream may have messed up and used macos_11_0 as the tag for the entire release instead of macos_11 and then did the same for subsequent releases. That issue will not be fixable since by now many downstream projects now rely the behavior of the incorrect tagging.

Delocate does not account for this and is trying to give an accurate platform tag which isn't supported by the upstream packaging. We could complain upstream but I think it's too late and Delocate should simply map all macos_X_Y tags after macos_11_0 to macos_X_0. This workaround can be undone whenever this gets resolved upstream.

Correct me if any of this is wrong.

[HexDecimal]

Rereading the posts on this issue seems to confirm my suspicions. It was an error to include the minor version number for the later versions of MacOS.

[rgommers]

As I reread pypa/packaging#578, it seems the upstream may have messed up and used macos_11_0 as the tag for the entire release instead of macos_11 and then did the same for subsequent releases. That issue will not be fixable since by now many downstream projects now rely the behavior of the incorrect tagging.

That is right. The better fix would have been to start allowing minor versions, but there's no movement on that.

Delocate does not account for this and is trying to give an accurate platform tag which isn't supported by the upstream packaging. We could complain upstream but I think it's too late and Delocate should simply map all macos_X_Y tags after macos_11_0 to macos_X_0. This workaround can be undone whenever this gets resolved upstream.

That also sounds like the right assessment. The output should be a wheel that's installable, so it must end in _0 for macOS>=11. The remaining question, is: should it be upgraded or downgraded by default? Probably the latter, since in the vast majority of cases, that will work and will be more compatible.

A prominent warning if a deployment target gets downgraded by delocate (e.g., 12_7 to 12_0) would be great though, because it's not always going to be correct. Example: macOS Accelerate was new in 13_3, so when we compile against it in NumPy and SciPy we actually have to manually rename the wheels to 14_0 and not 13_0 (13_0 would work on a >=13.3 build machine, but break users on 13.0-13.2 badly).

[HexDecimal]

The 2nd version number is truncated in a similar way as the third version number is truncated in macOS releases before 11. At least that's how I see it. The wheel tag is shown as macos_11_0 but you could think of it as macos_11.

Dependency version verification is already in, but you probably didn't notice it yet because it's also what caused this regression in the first place. Soon MACOSX_DEPLOYMENT_TARGET=13.2 will confirm that bundled libraries are compatible with 13.2 and then label the wheel as macos_13_0. Use MACOSX_DEPLOYMENT_TARGET=14.0 if you wanted to "upgrade" instead. Not meeting MACOSX_DEPLOYMENT_TARGET will be a strict error including a report of which libraries did not meet the requirements.

[rgommers]

Thanks. I understand all that. My point was that this:

will confirm that bundled libraries are compatible with 13.2 and then label the wheel as macos_13_0.

will be new, and it may yield wheels that will not work on 13.0 or 13.1. That requires a prominent warning I'd say.

[HexDecimal]

I'm not a native Mac user. I need to know more about the nature of the minor version in macOS 11+ to have an opinion on it. If it's simply a minor OS update for that release then it doesn't seem to be as big of a deal for casual deployment, tell devs to keep their OS up-to-date. On more stable systems where it does matter, I trust devs to be aware of what MACOSX_DEPLOYMENT_TARGET means without needing a warning about its repercussions.

The upstream packagers have decided that the platform tags for macOS refer to each macOS release rather than the actual version numbers. I'd like to include exact versions in the wheel metadata whenever they allow it.

Notes and advice on how one should use MACOSX_DEPLOYMENT_TARGET and about version compatibility edge cases would be good to add to the Delocate readme.

[agriyakhetarpal]

I need to know more about the nature of the minor version in macOS 11+ to have an opinion on it. If it's simply a minor OS update for that release then it doesn't seem to be as big of a deal for casual deployment, tell devs to keep their OS up-to-date.

In (rare) cases, packages rely on an absurd deployment target. For example, zig-pypi is one; it requires 11.7 because of the unavailability of a very low-level symbol in the XNU kernel. Not having support for minor macOS versions means that users need to upgrade to Monterey (12.0) altogether to use the package with pip/Python, when they could meanwhile run the frontend or even bootstrap the language on 11.X with or without providing the LIBC themselves.

[HexDecimal]

That's a pretty good example. In that case maybe a warning for when MACOSX_DEPLOYMENT_TARGET is not provided?

[agriyakhetarpal]

In that case maybe a warning for when MACOSX_DEPLOYMENT_TARGET is not provided?

I think so. However, if one were to use cibuildwheel which sets it automatically (to whichever value), and then delocate tries to repair it during the wheel repair stage, and if one provides a test command via CIBW_TEST_COMMAND, pip will fail to install it anyway – the error message from pip might not make it entirely clear and it isn't the best, but it at least does give indications that something is wrong. So I feel it's more about tying in and synchronising the workflow in these two scenarios: where delocate-wheel is used with cibuildwheel and outside it. To do this, emitting a warning about potential breakage if the CIBUILDWHEEL environment variable is set could be an option. It is to note that there might be (and are) authors that are not using CIBW_TEST_COMMAND to check the wheels they built (because it's not strictly needed) or in some occasions not using cibuildwheel altogether.

[HexDecimal]

I expect cibuildwheel will set a reasonable MACOSX_DEPLOYMENT_TARGET which probably won't pass the check in delocate-wheel in all but the most ideal circumstances. This just means that setting up cibuildwheel includes manually setting the MACOSX_DEPLOYMENT_TARGET variable which I had no problem adding to the env of the GitHub action.

Outside of cibuildwheel there are the main two options: Either without MACOSX_DEPLOYMENT_TARGET where Delocate will determine and use the lowest macOS release the wheel can partially support, or with MACOSX_DEPLOYMENT_TARGET which will set the wheels platform.

I'd suggest continuing this discussion on #219.