pypa
packaging.tags._platform_tags() does not report current OS version
Overview
On MacOS 11.2 arm64, the list returned by packaging.tags._platform_tags() does not contain macosx_11_2_arm64. The closest match is macosx_11_0_arm64.
Consequences
Because of this issue, pip does not recognize cached wheels that have been built on the same machine and instead repeats long compile processes. For example, it builds and caches numpy-1.20.3-cp39-cp39-macosx_11_2_arm64.whl but then does not recognize it as a compatible wheel and rebuilds it.
To Reproduce
>>> import sys
>>> sys.version
'3.9.5 (default, May 3 2021, 19:12:05) \n[Clang 12.0.5 (clang-1205.0.22.9)]'
>>> import platform
>>> platform.mac_ver()
('11.2.3', ('', '', ''), 'arm64')
>>> import packaging, packaging.tags
>>> packaging.__version__
'20.9'
>>> list(packaging.tags._platform_tags())
['macosx_11_0_arm64', 'macosx_11_0_universal2', 'macosx_10_16_universal2', 'macosx_10_15_universal2', 'macosx_10_14_universal2', 'macosx_10_13_universal2', 'macosx_10_12_universal2', 'macosx_10_11_universal2', 'macosx_10_10_universal2', 'macosx_10_9_universal2', 'macosx_10_8_universal2', 'macosx_10_7_universal2', 'macosx_10_6_universal2', 'macosx_10_5_universal2', 'macosx_10_4_universal2']
It seems like all 11.x are being normalised to 11.0 explicitly:
https://github.com/pypa/packaging/blob/73507464faca545d83b748929e60ec67c0cb50b1/packaging/tags.py#L402-L411
Since this code is contributed by an Apple employee, I feel we should ask Apple for some clarification on this. @lawrence-danna-apple
Who is generating that wheel file name?
If my memory serves me, the flooring is because we have no idea what the maximum potential minor version of macOS is, and builds should be compatible for any major version of the OS. So if you have a wheel built on macOS 11.987 but you're running macOS 11.2, it should still work, but there's no way to really express that short of dropping the minor version number for macOS > 11.
In this case, pip is generating the filename. I assume that is via the packaging library. I can try to trace it more fully if needed. I might need some guidance on that.
So are you getting the file name from pip and then naming your wheel file yourself? Pip is not a build tool (e.g. setuptools and wheel are a build tool), so I'm not quite following how pip is participating in the wheel file naming.
Not saying it’s the case here, but shouldn’t developers be able to have legistimate reasons to require at least 11.2 and later? And if they do, the resulting wheel should be installable on 11.2+. So packaging.tags should generate 11.2 tags (and downwards) on macOS 11.2 to signify the platform is compatible with those 11.2-and-later-only wheels. The flooring should happen in the build tool instead (i.e. given a list of possible tags provided by packaging.tags, choose the one that has the best compatibility given the build-time constraints).
@brettcannon No, the wheel is being built during a pip install and then cached with the filename containing macosx_11_2. When I try to install the same package version, pip does not use the cached wheel, and this seems to be because platform_tags as reported by packaging only contains macosx_11_0. Or at least that's my interpretation. I could be wrong. This wouldn't be such a big idea except for packages like Numpy can take many minutes to rebuild.
@uranusjr Maybe, but I believe @lawrence-danna-apple did this on purpose as macOS is making a promise that all 11.x wheels will be compatible. I don't have an issue working backwards for compatibility on a platform, but we should probably give @lawrence-danna-apple a chance to reply before we move forward with any potential change.
@DanLipsitt if this is about numpy then the build tool will be (I believe) their custom setuptools setup plus wheel (pip is just calling those tools on numpy's behalf). So it's probably wheel that's making the macOS 11.2 wheel.
@brettcannon It happens with opencv-python too. I forgot to mention that I built them with --no-use-pep517 because I couldn't get them to build without that flag on Apple Silicon.
I took a look at wheel and it uses a vendored packaging that returns this list of tags:
>>> from wheel.vendored.packaging import tags
>>> list(tags._platform_tags())
['macosx_11_0_arm64', 'macosx_11_0_universal2']
So that's not where macosx_11_2 is coming from.
I will take a closer look at how the packages are being built.
@DanLipsitt did you happen to ever figure out why your builds are getting the macosx_11_2 platform tag?
@uranusjr since we have not heard from @lawrence-danna-apple I'm going to ask @ronaldoussoren and @ned-deily if they have an opinion about changing the logic. Otherwise I would loathe to assume I knew macOS better than an Apple employee who added that logic on purpose and change how it has been working.
Yeah, there are definitely some issues here. What's driving this is that Apple changed macOS versioning as of macOS 11 (Big Sur), going from a three-level versioning system in previous macOS releases (similar to the scheme used by cPython) to a two-level one (similar to what Apple's iOS has been using). At the cPython level, we didn't quite grasp all the implications of that (other non-Python projects have been impacted by this as well) and I think we are still not handling things quite right. That also has an impact on packaging. I will look at this further and report back here.
I don't have the bandwidth to look into this, but I agree with what Ned writes.
It might be better to bite the bullet and switch to using just the major version for macOS 11 and later, that is "macosx_11_universal2".
Apple can introduce new functionality in minor releases (such as macOS 10.15.3 or macOS 11.1), but so far I've never seen such changes in unix-level APIs. Frameworks do changes at times (new AppKit APIs for example), but those should affect only a small number of projects and those should handle weak-linking such new APIs anyway. For a project as numpy the wheel generated on macOS 11.2 should work fine on macOS 11.0 without such shenanigans.
P.S. Please prefer universal2 wheels over singel architecture wheels when installing ;-)
If we have to special-case macOS 11 to emit both 11_0 and 11 as a version and then switch to major-only that's not a big deal (there's already so much special-casing due to architectural switches for macOS one more won't kill us 😉 ).
@ronaldoussoren I see your push for #381 😉
To state the obvious: changing from 11_0 to 11 would require changes to package building tools (in this particular case likely setuptools).
Fixing this on the pip side would require changing the list of compatible wheels. This already contains a long list of versions back to macOS 10.4, adding subreleases of macOS 11 to the list wouldn't be too bad. That said, those subreleases being bug fix releases does complicate thing as you wouldn't want to hardcode the latest bug fix release in every major release (that is, I'm on macOS 12.0, which 11.x wheels are considered compatible). I guess just assuming that those minor releases are single digit would be good enough (tm), Mac OS 10.4 was the last release that required 2 digits (10.4.10) and that was in a time before yearly releases.