Missing valid mac tags

[FFY00]

PEP 425 says (https://peps.python.org/pep-0425/#platform-tag)

The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _.

On recent mac versions

>>> distutils.util.get_platform()
'macosx-12-arm64'

However, pypa/packaging does not consider macosx_12_arm64 a valid tag, only macosx_12_0_arm64

>>> list(packaging.tags.mac_platforms())
['macosx_12_0_arm64',
 'macosx_12_0_universal2',
 'macosx_11_0_arm64',
 'macosx_11_0_universal2',
 'macosx_10_16_universal2',
 'macosx_10_15_universal2',
 'macosx_10_14_universal2',
 'macosx_10_13_universal2',
 'macosx_10_12_universal2',
 'macosx_10_11_universal2',
 'macosx_10_10_universal2',
 'macosx_10_9_universal2',
 'macosx_10_8_universal2',
 'macosx_10_7_universal2',
 'macosx_10_6_universal2',
 'macosx_10_5_universal2',
 'macosx_10_4_universal2']

[brettcannon]

Do we know if that tag makes sense?

[FFY00]

Can you clarify? The tag seems fine to me, just missing the minor version.

[uranusjr]

The tag makes sense. Apple used to use 10.x (major dot minor), but recently starts to use only one single number to version the platform (11, 12 onward). 11.0 and 12.0 are arguably not valid, but we might want to keep them for compatibility since currently existing wheel generators may be using them now.

[FFY00]

That's great to know! I don't use mac so I have not kept up to date with the versioning scheme.

And I agree that we probably want to keep this old tags valid.

I will update the PR when I am back to the computer. The proposed change would be for 11 and 12 generate both version, with and without the minor version, and for 13 and onwards only generate tags with the major version. Does that sound good?

[uranusjr]

Yeah, I guess that makes sense. Apple never tells anyone how they do things and it’s impossible to know if the version scheme is going to change again, but the strategy you outlined is likely the best we can do.

Another issue not related to packaging but wheel installers: When macOS 11 was announced, Apple did mention the possibility of having minor versions. Would a hypothetical macOS 12.1 correctly match macosx_12_universal2?

[FFY00]

Well, the PEP does not specify anything about matching older versions, it just maps the platform tag to distutils.util.get_platform, but given the current implementations, I'd say yes. I think the easiest thing to do is kinda interpret it as 12.0, or rather, 12 comes before 12.1.

[FFY00]

Also, I think it would make sense to update the PEP to say to use sysconfig.get_platform when distutils is not available. Any thoughts?

[brettcannon]

When macOS 11 was announced, Apple did mention the possibility of having minor versions. Would a hypothetical macOS 12.1 correctly match macosx_12_universal2?

That's what my question was about; does a minor-version-free macOS version make sense?

I think the easiest thing to do is kinda interpret it as 12.0, or rather, 12 comes before 12.1.

What would happen is we list all the appropriate tags in preference order, and then tools can use that order however they choose. So in this instance we would probably list macosx_12_universal2 after all the other macosx_12_*_universal2 wheels since it's the least specific.

I think it would make sense to update the PEP to say to use sysconfig.get_platform when distutils is not available. Any thoughts?

We should probably migrate that entire PEP to packaging.python.org. But in this specific case I don't know what makes the most sense since the tag creation logic is rather complicated and not well-specified already, so I don't know if continuing to specify any part of it is important.

[FFY00]

What would happen is we list all the appropriate tags in preference order, and then tools can use that order however they choose. So in this instance we would probably list macosx_12_universal2 after all the other macosx_12_*_universal2 wheels since it's the least specific.

In this specific case, I think we should probably list it earlier, and basically treat it as an alternative to macosx_12_0_universal2, given that we probably shouldn't be generating macosx_12_0_universal2 in the first place. Basically, macosx_12_0_universal2 would be an undesirable alias to macosx_12_universal2. Does that make sense?

We should probably migrate that entire PEP to packaging.python.org.

Agreed.

But in this specific case I don't know what makes the most sense since the tag creation logic is rather complicated and not well-specified already, so I don't know if continuing to specify any part of it is important.

I don't know how you would define this field in that case.

[uranusjr]

That's what my question was about; does a minor-version-free macOS version make sense?

It does in the sense that macOS does return a minor-less 12. So it makes sense for a wheel generator to use this version scheme when creating a wheel. The problem here is though, are our installer implementations smart enough to correctly detect compatibility if a wheel with minor-less version is being installed on a platform that does contain a minor segment. This is where specs are vague; PEPs don’t specify how platform version compatibility should be detected, and Apple doesn’t say if it’s ever possible to have a minor version segment in the future, and how that minor version bump would affect compatibility.

[rgommers]

Apple used to use 10.x (major dot minor), but recently starts to use only one single number to version the platform (11, 12 onward).

This seems incorrect, the current macOS version is 12.4:

11.0 and 12.0 are arguably not valid, but we might want to keep them for compatibility since currently existing wheel generators may be using them now.

These version numbers do not depend directly on macOS, they're generated by Python so it depends where you install Python from. On macOS 12.4 with Python 3.9.6 from conda-forge, I get:

>>> distutils.util.get_platform()
'macosx-11.0-arm64'
>>> sysconfig.get_platform()
'macosx-11.0-arm64'

The original bug report on https://github.com/FFY00/meson-python/issues/91 came from a build with Python 3.8 from Homebrew.

The Python shipped by Apple (/usr/bin/python3) actually gives 10.14:

>>> os.__file__
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py'
>>> import distutils.util
>>> distutils.util.get_platform()
'macosx-10.14-arm64'

I suggest to simply add the missing tags and make no other changes.

[FFY00]

Another issue: https://github.com/FFY00/meson-python/issues/160

The minor versions are also not generated. Should I be generating 12.0 tags for 12.6? That seems like the only option right now to make it work with the current packaging code.

[brettcannon]

Should I be generating 12.0 tags for 12.6?

I think so. I believe the last time we discussed how to do version number support for the "new" macOS versions the guidance this was what was suggested. We can discuss doing something different (e.g. all versions, no minor number), but I think the concern with the latter was whether anyone made weird assumptions about always having a minor number.

But my memory may be faulty with all of this, so reading the old issues is the best way to double-check me.

[uranusjr]

Yes I think from previous discussions, a wheel against 12.0 should be installable on 12.6.

[rgommers]

That is all correct - and yes, we should generate 12_0 tags right now for them to be installable.

That said, there is no reason for packaging to reject 12_6 wheels for users who are on >= 12.6. And it is possible for packages to need fixes that are part of 12.6 but not 12.0. In that case, they want to set MACOSX_DEPLOYMENT_TARGET=12.6 and produce wheels containing the 12_6 tag. So packaging needs to be updated to accept such tags - in addition to the plain 12 major-version-only tag.

[brettcannon]

Now I remember why we didn't want to do midyear updates in the tags; we don't know how to back-fill older macOS versions without hard-coding the final minor version number. For instance, while we can infer 12.6 to 12.0, what do we do for macOS 11? We don't know what the upper limit is for a minor version until Apple announces that version is EOL. So until that happens we can only guess what the upper bound is for macOS 11 (e.g. is it 11.5, 11.10, 11.100?). So either we have to hard-code the upper limit known at any moment in time and then push a new version of packaging every time Apple releases a new version of macOS, set some crazy upper limit we think they will never go past, or do what we do and only set the lower-bound of *.0.

[rgommers]

I don't really see why that is required - why wouldn't you just say that any 11_x (or in general X_Y with X and Y integers) tag is valid? There's no practical problems with that that I can think of, it's just a valid PEP 3149 compliant tag. Analogy: manylinux tags are valid for any glibc major_minor version: https://peps.python.org/pep-0600/#specification.

[brettcannon]

I don't really see why that is required - why wouldn't you just say that any 11_x (or in general X_Y with X and Y integers) tag is valid?

Potentially, but then why do we care about 12_x? If we are concerned about a change in 12.6 that is only forward-compatible, then we can't assume 12.0 is okay once 13.0 comes out for some feature. And since we list every compatible wheel tag, we can't really make the declaration that, "macOS 12.6 is compatible with the 12_0/12 tag when running under macOS 13," without listing 12_6 explicitly.

Or put another way (to make sure I understand), if you're running macOS 13.2 and have a wheel that wants 12_6, how do you handle that when the wheel tags you have are:

• 13_2
• 13_1
• 13_0
• 12/12_0

without making every consumer of packaging.tags specify handling macOS platform tags?

Analogy: manylinux tags are valid for any glibc major_minor version: https://peps.python.org/pep-0600/#specification.

That would be true if the glibc team told us they have no plans to ever change the major version number, but they said they didn't when that discussion came up. Otherwise we will have a similar problem. But glibc also bumps their major version number way less than macOS, so hard-coding whatever glibc 2 stops at is much less effort than trying to keep up with macOS' release cadence.

[rgommers]

Potentially, but then why do we care about 12_x?

Aside from possible incompatibilities - which could indeed occur - because it makes the tags easier to understand and specify. These issues came up to begin with because we implemented tag support in meson-python and then ran into pip not understanding 12_6. It's entirely logical to think that if you can set a deployment target explicitly to 10.9, you should also be able to set it to 12.6 (via the MACOSX_DEPLOYMENT_TARGET). Right now, what is a backend to do if a user does that? Explicitly error out? Or printing a warning message and then resetting it to 12.0?

It's the same for 11 rather than 11_0 as a tag. Right now this is all simply undocumented stuff that bites you unless you read the packaging source code, for what looks like valid tags. So it's be nice to be permissive rather than overly strict.

Potentially, but then why do we care about 12_x? If we are concerned about a change in 12.6 that is only forward-compatible, then we can't assume 12.0 is okay once 13.0 comes out for some feature

It's a major version number change - I think there are no watertight guarantees about 13.0 either way. But besides that, I don't think this argument holds. In corner cases 12.6 can be a breaking release, and so can 13.0. The latter is more likely, but both are possible. For 99.9x% of wheels there will be no problem (because the breakage will likely be for something obscure), so you don't have to change today's assumptions.

so hard-coding whatever glibc 2 stops at is much less effort

I may be missing something here. Is that actually being done anywhere?

[rgommers]

To make the case for an ability to set macOS deployment target more concrete, here is a real-world case:

• SciPy sets the deployment target for all its macOS arm64 wheels to 12.0 (by manually renaming them, we have to build on macOS 11 CI runners right now)
• The reason is that a kernel panic (which is way worse than a regular crash) may happen due to a bug in older macOS system libraries, which is triggered by some linear algebra functionality
• This bug happens to be fixed from macOS 12.0.1 onwards (relevant issue comment), and the fix could not be backported. nor was it clear which exact system library was responsible.
• We want to be able to set the deployment target to the minimum version with the fix. Right now 12_0 isn't too bad, we're just ignoring that 12.0.0 is not working (not many users will have that version, so it's okay in practice).
  • Clearly, if the bug fix had landed in say 12.5, we'd want 12_5 - and 12_0 would be pretty horrible.

So this has nothing to do with "will it run on 13.0 or not" (it will, unless Apple happens to break something unrelated in the future), but with "the older OS version has an unpatched bug that we need to avoid".

Or put another way (to make sure I understand), if you're running macOS 13.2 and have a wheel that wants 12_6, how do you handle that when the wheel tags you have are:

* `13_2`
* `13_1`
* `13_0`
* `12`/`12_0`

• When you are building a wheel, you honor the explicitly requested deployment target of 12.6 and the resulting wheel tag will be 12_6.
• At install time, you install it if the OS version is >=12.6 (which includes all 13.x, but not 12.0).

without making every consumer of packaging.tags specify handling macOS platform tags?

I don't understand this part of your question. You have to know less. All you have to do is use a major_minor version scheme. Right now you have to know that that is not going to work because of packaging, and explicitly replace the minor version part with .0 .

You seem to be working under the assumption that the packaging tooling must know whether 12.6 was actually released by Apple or not. But why is this relevant at all? As far as I can tell, nothing will go wrong if it doesn't.

[brettcannon]

You seem to be working under the assumption that the packaging tooling must know whether 12.6 was actually released by Apple or not. But why is this relevant at all? As far as I can tell, nothing will go wrong if it doesn't.

Because I don't know where we would need to stop generating those tags.

To make this concrete, if you are on macOS 13, what tags would you have us generate to cover macOS 12?

[rgommers]

Because I don't know where we would need to stop generating those tags.

We do not need an explicit list of all valid tags for either tag generation at build time or tag validation at install time (see the bullet points in my comment above). I think the only reason you are asking is because that is what packaging.tags.mac_platforms currently does. Is that right?

Unless I'm missing a use case here, a better design would be to have a function that validates a given tag given the target interpreter, rather than "return a list of all valid tags for this platform" and then have pip & co. just use that to check if the tag they are seeing is in that huge list of valid tags.

To make this concrete, if you are on macOS 13, what tags would you have us generate to cover macOS 12?

If you don't want to change what mac_platforms currently does, just pick a minor version number that's high enough. Say generate 12_0 ... 12_42. It really shouldn't matter, since they are all valid major_minor version numbers.

It also doesn't seem to matter for performance by the way. I checked what pip currently uses (a 12_6 tag will fail at https://github.com/pypa/pip/blob/0d4e9eb72253c008f2790482e664ce92198c5240/src/pip/_internal/resolution/resolvelib/factory.py#L131-L138), and the list of tags it generates to compare against is already huge:

ipdb> for tag in self._finder.target_python.get_tags():
    print(tag)

cp39-cp39-macosx_12_0_arm64
cp39-cp39-macosx_12_0_universal2
cp39-cp39-macosx_11_0_arm64
cp39-cp39-macosx_11_0_universal2
cp39-cp39-macosx_10_16_universal2
cp39-cp39-macosx_10_15_universal2
cp39-cp39-macosx_10_14_universal2
cp39-cp39-macosx_10_13_universal2
cp39-cp39-macosx_10_12_universal2
cp39-cp39-macosx_10_11_universal2
cp39-cp39-macosx_10_10_universal2
cp39-cp39-macosx_10_9_universal2
cp39-cp39-macosx_10_8_universal2
cp39-cp39-macosx_10_7_universal2
cp39-cp39-macosx_10_6_universal2
cp39-cp39-macosx_10_5_universal2
cp39-cp39-macosx_10_4_universal2
cp39-abi3-macosx_12_0_arm64
cp39-abi3-macosx_12_0_universal2
cp39-abi3-macosx_11_0_arm64
cp39-abi3-macosx_11_0_universal2
cp39-abi3-macosx_10_16_universal2
cp39-abi3-macosx_10_15_universal2
cp39-abi3-macosx_10_14_universal2
cp39-abi3-macosx_10_13_universal2
cp39-abi3-macosx_10_12_universal2
cp39-abi3-macosx_10_11_universal2
cp39-abi3-macosx_10_10_universal2
cp39-abi3-macosx_10_9_universal2
cp39-abi3-macosx_10_8_universal2
cp39-abi3-macosx_10_7_universal2
cp39-abi3-macosx_10_6_universal2
cp39-abi3-macosx_10_5_universal2
cp39-abi3-macosx_10_4_universal2
cp39-none-macosx_12_0_arm64
cp39-none-macosx_12_0_universal2
cp39-none-macosx_11_0_arm64
cp39-none-macosx_11_0_universal2
cp39-none-macosx_10_16_universal2
cp39-none-macosx_10_15_universal2
cp39-none-macosx_10_14_universal2
cp39-none-macosx_10_13_universal2
cp39-none-macosx_10_12_universal2
cp39-none-macosx_10_11_universal2
cp39-none-macosx_10_10_universal2
cp39-none-macosx_10_9_universal2
cp39-none-macosx_10_8_universal2
cp39-none-macosx_10_7_universal2
cp39-none-macosx_10_6_universal2
cp39-none-macosx_10_5_universal2
cp39-none-macosx_10_4_universal2
cp38-abi3-macosx_12_0_arm64
cp38-abi3-macosx_12_0_universal2
cp38-abi3-macosx_11_0_arm64
cp38-abi3-macosx_11_0_universal2
cp38-abi3-macosx_10_16_universal2
cp38-abi3-macosx_10_15_universal2
cp38-abi3-macosx_10_14_universal2
cp38-abi3-macosx_10_13_universal2
cp38-abi3-macosx_10_12_universal2
cp38-abi3-macosx_10_11_universal2
cp38-abi3-macosx_10_10_universal2
cp38-abi3-macosx_10_9_universal2
cp38-abi3-macosx_10_8_universal2
cp38-abi3-macosx_10_7_universal2
cp38-abi3-macosx_10_6_universal2
cp38-abi3-macosx_10_5_universal2
cp38-abi3-macosx_10_4_universal2
cp37-abi3-macosx_12_0_arm64
cp37-abi3-macosx_12_0_universal2
cp37-abi3-macosx_11_0_arm64
cp37-abi3-macosx_11_0_universal2
cp37-abi3-macosx_10_16_universal2
cp37-abi3-macosx_10_15_universal2
cp37-abi3-macosx_10_14_universal2
cp37-abi3-macosx_10_13_universal2
cp37-abi3-macosx_10_12_universal2
cp37-abi3-macosx_10_11_universal2
cp37-abi3-macosx_10_10_universal2
cp37-abi3-macosx_10_9_universal2
cp37-abi3-macosx_10_8_universal2
cp37-abi3-macosx_10_7_universal2
cp37-abi3-macosx_10_6_universal2
cp37-abi3-macosx_10_5_universal2
cp37-abi3-macosx_10_4_universal2
cp36-abi3-macosx_12_0_arm64
cp36-abi3-macosx_12_0_universal2
cp36-abi3-macosx_11_0_arm64
cp36-abi3-macosx_11_0_universal2
cp36-abi3-macosx_10_16_universal2
cp36-abi3-macosx_10_15_universal2
cp36-abi3-macosx_10_14_universal2
cp36-abi3-macosx_10_13_universal2
cp36-abi3-macosx_10_12_universal2
cp36-abi3-macosx_10_11_universal2
cp36-abi3-macosx_10_10_universal2
cp36-abi3-macosx_10_9_universal2
cp36-abi3-macosx_10_8_universal2
cp36-abi3-macosx_10_7_universal2
cp36-abi3-macosx_10_6_universal2
cp36-abi3-macosx_10_5_universal2
cp36-abi3-macosx_10_4_universal2
cp35-abi3-macosx_12_0_arm64
cp35-abi3-macosx_12_0_universal2
cp35-abi3-macosx_11_0_arm64
cp35-abi3-macosx_11_0_universal2
cp35-abi3-macosx_10_16_universal2
cp35-abi3-macosx_10_15_universal2
cp35-abi3-macosx_10_14_universal2
cp35-abi3-macosx_10_13_universal2
cp35-abi3-macosx_10_12_universal2
cp35-abi3-macosx_10_11_universal2
cp35-abi3-macosx_10_10_universal2
cp35-abi3-macosx_10_9_universal2
cp35-abi3-macosx_10_8_universal2
cp35-abi3-macosx_10_7_universal2
cp35-abi3-macosx_10_6_universal2
cp35-abi3-macosx_10_5_universal2
cp35-abi3-macosx_10_4_universal2
cp34-abi3-macosx_12_0_arm64
cp34-abi3-macosx_12_0_universal2
cp34-abi3-macosx_11_0_arm64
cp34-abi3-macosx_11_0_universal2
cp34-abi3-macosx_10_16_universal2
cp34-abi3-macosx_10_15_universal2
cp34-abi3-macosx_10_14_universal2
cp34-abi3-macosx_10_13_universal2
cp34-abi3-macosx_10_12_universal2
cp34-abi3-macosx_10_11_universal2
cp34-abi3-macosx_10_10_universal2
cp34-abi3-macosx_10_9_universal2
cp34-abi3-macosx_10_8_universal2
cp34-abi3-macosx_10_7_universal2
cp34-abi3-macosx_10_6_universal2
cp34-abi3-macosx_10_5_universal2
cp34-abi3-macosx_10_4_universal2
cp33-abi3-macosx_12_0_arm64
cp33-abi3-macosx_12_0_universal2
cp33-abi3-macosx_11_0_arm64
cp33-abi3-macosx_11_0_universal2
cp33-abi3-macosx_10_16_universal2
cp33-abi3-macosx_10_15_universal2
cp33-abi3-macosx_10_14_universal2
cp33-abi3-macosx_10_13_universal2
cp33-abi3-macosx_10_12_universal2
cp33-abi3-macosx_10_11_universal2
cp33-abi3-macosx_10_10_universal2
cp33-abi3-macosx_10_9_universal2
cp33-abi3-macosx_10_8_universal2
cp33-abi3-macosx_10_7_universal2
cp33-abi3-macosx_10_6_universal2
cp33-abi3-macosx_10_5_universal2
cp33-abi3-macosx_10_4_universal2
cp32-abi3-macosx_12_0_arm64
cp32-abi3-macosx_12_0_universal2
cp32-abi3-macosx_11_0_arm64
cp32-abi3-macosx_11_0_universal2
cp32-abi3-macosx_10_16_universal2
cp32-abi3-macosx_10_15_universal2
cp32-abi3-macosx_10_14_universal2
cp32-abi3-macosx_10_13_universal2
cp32-abi3-macosx_10_12_universal2
cp32-abi3-macosx_10_11_universal2
cp32-abi3-macosx_10_10_universal2
cp32-abi3-macosx_10_9_universal2
cp32-abi3-macosx_10_8_universal2
cp32-abi3-macosx_10_7_universal2
cp32-abi3-macosx_10_6_universal2
cp32-abi3-macosx_10_5_universal2
cp32-abi3-macosx_10_4_universal2
py39-none-macosx_12_0_arm64
py39-none-macosx_12_0_universal2
py39-none-macosx_11_0_arm64
py39-none-macosx_11_0_universal2
py39-none-macosx_10_16_universal2
py39-none-macosx_10_15_universal2
py39-none-macosx_10_14_universal2
py39-none-macosx_10_13_universal2
py39-none-macosx_10_12_universal2
py39-none-macosx_10_11_universal2
py39-none-macosx_10_10_universal2
py39-none-macosx_10_9_universal2
py39-none-macosx_10_8_universal2
py39-none-macosx_10_7_universal2
py39-none-macosx_10_6_universal2
py39-none-macosx_10_5_universal2
py39-none-macosx_10_4_universal2
py3-none-macosx_12_0_arm64
py3-none-macosx_12_0_universal2
py3-none-macosx_11_0_arm64
py3-none-macosx_11_0_universal2
py3-none-macosx_10_16_universal2
py3-none-macosx_10_15_universal2
py3-none-macosx_10_14_universal2
py3-none-macosx_10_13_universal2
py3-none-macosx_10_12_universal2
py3-none-macosx_10_11_universal2
py3-none-macosx_10_10_universal2
py3-none-macosx_10_9_universal2
py3-none-macosx_10_8_universal2
py3-none-macosx_10_7_universal2
py3-none-macosx_10_6_universal2
py3-none-macosx_10_5_universal2
py3-none-macosx_10_4_universal2
py38-none-macosx_12_0_arm64
py38-none-macosx_12_0_universal2
py38-none-macosx_11_0_arm64
py38-none-macosx_11_0_universal2
py38-none-macosx_10_16_universal2
py38-none-macosx_10_15_universal2
py38-none-macosx_10_14_universal2
py38-none-macosx_10_13_universal2
py38-none-macosx_10_12_universal2
py38-none-macosx_10_11_universal2
py38-none-macosx_10_10_universal2
py38-none-macosx_10_9_universal2
py38-none-macosx_10_8_universal2
py38-none-macosx_10_7_universal2
py38-none-macosx_10_6_universal2
py38-none-macosx_10_5_universal2
py38-none-macosx_10_4_universal2
py37-none-macosx_12_0_arm64
py37-none-macosx_12_0_universal2
py37-none-macosx_11_0_arm64
py37-none-macosx_11_0_universal2
py37-none-macosx_10_16_universal2
py37-none-macosx_10_15_universal2
py37-none-macosx_10_14_universal2
py37-none-macosx_10_13_universal2
py37-none-macosx_10_12_universal2
py37-none-macosx_10_11_universal2
py37-none-macosx_10_10_universal2
py37-none-macosx_10_9_universal2
py37-none-macosx_10_8_universal2
py37-none-macosx_10_7_universal2
py37-none-macosx_10_6_universal2
py37-none-macosx_10_5_universal2
py37-none-macosx_10_4_universal2
py36-none-macosx_12_0_arm64
py36-none-macosx_12_0_universal2
py36-none-macosx_11_0_arm64
py36-none-macosx_11_0_universal2
py36-none-macosx_10_16_universal2
py36-none-macosx_10_15_universal2
py36-none-macosx_10_14_universal2
py36-none-macosx_10_13_universal2
py36-none-macosx_10_12_universal2
py36-none-macosx_10_11_universal2
py36-none-macosx_10_10_universal2
py36-none-macosx_10_9_universal2
py36-none-macosx_10_8_universal2
py36-none-macosx_10_7_universal2
py36-none-macosx_10_6_universal2
py36-none-macosx_10_5_universal2
py36-none-macosx_10_4_universal2
py35-none-macosx_12_0_arm64
py35-none-macosx_12_0_universal2
py35-none-macosx_11_0_arm64
py35-none-macosx_11_0_universal2
py35-none-macosx_10_16_universal2
py35-none-macosx_10_15_universal2
py35-none-macosx_10_14_universal2
py35-none-macosx_10_13_universal2
py35-none-macosx_10_12_universal2
py35-none-macosx_10_11_universal2
py35-none-macosx_10_10_universal2
py35-none-macosx_10_9_universal2
py35-none-macosx_10_8_universal2
py35-none-macosx_10_7_universal2
py35-none-macosx_10_6_universal2
py35-none-macosx_10_5_universal2
py35-none-macosx_10_4_universal2
py34-none-macosx_12_0_arm64
py34-none-macosx_12_0_universal2
py34-none-macosx_11_0_arm64
py34-none-macosx_11_0_universal2
py34-none-macosx_10_16_universal2
py34-none-macosx_10_15_universal2
py34-none-macosx_10_14_universal2
py34-none-macosx_10_13_universal2
py34-none-macosx_10_12_universal2
py34-none-macosx_10_11_universal2
py34-none-macosx_10_10_universal2
py34-none-macosx_10_9_universal2
py34-none-macosx_10_8_universal2
py34-none-macosx_10_7_universal2
py34-none-macosx_10_6_universal2
py34-none-macosx_10_5_universal2
py34-none-macosx_10_4_universal2
py33-none-macosx_12_0_arm64
py33-none-macosx_12_0_universal2
py33-none-macosx_11_0_arm64
py33-none-macosx_11_0_universal2
py33-none-macosx_10_16_universal2
py33-none-macosx_10_15_universal2
py33-none-macosx_10_14_universal2
py33-none-macosx_10_13_universal2
py33-none-macosx_10_12_universal2
py33-none-macosx_10_11_universal2
py33-none-macosx_10_10_universal2
py33-none-macosx_10_9_universal2
py33-none-macosx_10_8_universal2
py33-none-macosx_10_7_universal2
py33-none-macosx_10_6_universal2
py33-none-macosx_10_5_universal2
py33-none-macosx_10_4_universal2
py32-none-macosx_12_0_arm64
py32-none-macosx_12_0_universal2
py32-none-macosx_11_0_arm64
py32-none-macosx_11_0_universal2
py32-none-macosx_10_16_universal2
py32-none-macosx_10_15_universal2
py32-none-macosx_10_14_universal2
py32-none-macosx_10_13_universal2
py32-none-macosx_10_12_universal2
py32-none-macosx_10_11_universal2
py32-none-macosx_10_10_universal2
py32-none-macosx_10_9_universal2
py32-none-macosx_10_8_universal2
py32-none-macosx_10_7_universal2
py32-none-macosx_10_6_universal2
py32-none-macosx_10_5_universal2
py32-none-macosx_10_4_universal2
py31-none-macosx_12_0_arm64
py31-none-macosx_12_0_universal2
py31-none-macosx_11_0_arm64
py31-none-macosx_11_0_universal2
py31-none-macosx_10_16_universal2
py31-none-macosx_10_15_universal2
py31-none-macosx_10_14_universal2
py31-none-macosx_10_13_universal2
py31-none-macosx_10_12_universal2
py31-none-macosx_10_11_universal2
py31-none-macosx_10_10_universal2
py31-none-macosx_10_9_universal2
py31-none-macosx_10_8_universal2
py31-none-macosx_10_7_universal2
py31-none-macosx_10_6_universal2
py31-none-macosx_10_5_universal2
py31-none-macosx_10_4_universal2
py30-none-macosx_12_0_arm64
py30-none-macosx_12_0_universal2
py30-none-macosx_11_0_arm64
py30-none-macosx_11_0_universal2
py30-none-macosx_10_16_universal2
py30-none-macosx_10_15_universal2
py30-none-macosx_10_14_universal2
py30-none-macosx_10_13_universal2
py30-none-macosx_10_12_universal2
py30-none-macosx_10_11_universal2
py30-none-macosx_10_10_universal2
py30-none-macosx_10_9_universal2
py30-none-macosx_10_8_universal2
py30-none-macosx_10_7_universal2
py30-none-macosx_10_6_universal2
py30-none-macosx_10_5_universal2
py30-none-macosx_10_4_universal2
cp39-none-any
py39-none-any
py3-none-any
py38-none-any
py37-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any

[brettcannon]

We do not need an explicit list of all valid tags for either tag generation at build time or tag validation at install time (see the bullet points in my comment above). I think the only reason you are asking is because that is what packaging.tags.mac_platforms currently does. Is that right?

I ask because the library is designed to generate the list of valid tags, not to ask the library if the current system is valid for a specific tag.

Unless I'm missing a use case here, a better design would be to have a function that validates a given tag given the target interpreter, rather than "return a list of all valid tags for this platform" and then have pip & co. just use that to check if the tag they are seeing is in that huge list of valid tags.

The specs around tags are not structured to require asking a library if something is valid; it's meant to work against a static list of tags you were given. You're meant to be able to take a list of wheel tags and a list of platform tags and figure out what to use w/o using packaging. A JSON dump of what a platform supports should be an acceptable input to a tool resolving which wheel tags are acceptable; think of a SaaS provider giving you such a JSON file to let you ship the appropriate wheels/code to work on their service.

Another thing to point out is we have gone this long w/o finer granularity than the annual release version and basically been fine (i.e. all we never listed the micro releases from the 10.x version range).

I think we're at enough of an impasse that bringing this up on discuss.python.org is appropriate.

[rgommers]

Another thing to point out is we have gone this long w/o finer granularity than the annual release version and basically been fine (i.e. all we never listed the micro releases from the 10.x version range).

Yes agreed. I think we typically ignore the corner cases, and in practice it won't be so bad. Similar to Python releases, it's very common to deal with minor releases, but very rare to worry about a 3.x.0 release having a bug and therefore needing a wheel to only work on .1 and up. I guess when it does come up, folks just raise an exception at runtime with instructions to upgrade.

I think we're at enough of an impasse that bringing this up on discuss.python.org is appropriate.

If you don't mind, I'll leave it at this. I've already spent more time on investigating this than I had available, and moving this to Discourse is unlikely to bring a quick resolution. So I'll leave it to the next person with a real need for this.

[uranusjr]

Can we design the interface to provide something like is_tag_compatible instead? I’ve always felt generating a full list of tags cumbersome, but assumed there are reasons for that.

[brettcannon]

Can we design the interface to provide something like is_tag_compatible instead?

We could have a convenience function for that which simply does the logic of searching the list of tags. But you will probably also want a utility function which takes a set of wheel tags (or file names) and then returns which tag matches "the best" (i.e. highest in the list of supported tags). This would also let you define "best" in either terms of fastest or most compatible/generic based on the order of the list.

I’ve always felt generating a full list of tags cumbersome, but assumed there are reasons for that.

Well, the API is based on generators so you don't have to generate the full list if you get an answer early enough. 😉 But yes, the reason for the regeneration has been that a platform effectively states what tags it supports; we just happens to be calculated constantly because CPython and other interpreters don't record these details externally in some JSON file (which is something I have thought about).

[pradyunsg]

IIUC, what @uranusjr and @rgommers are suggesting is that we avoid the need to generate all the tags to determine compatibility with the current platform in a new API -- have the ability to validate the tag without needing to generate the entire itertools.product(pyvers, abis, platforms) matrix.

For example, instead of generating the entire sequence of tags in mac_platforms, having the ability to do has_compatible_mac_platform(tag) that's conditional instead of any(tag == incoming for incoming in mac_platforms()), looking something like...

def has_compatible_mac_platform(tag):
    parts = tag.platform.split("_")

    if len(parts) != 4 or parts[0] != "maxosx":
        return False

    major, minor, binary_format = int(parts[1]), int(parts[2]), parts[3]
    version_str, _, cpu_arch = platform.mac_ver()
    version = cast("MacVersion", tuple(map(int, version_str.split(".")[:2])))
    arch = _mac_arch(cpu_arch)

    # Check version
    if (major, minor) > version:
        return False
    # TODO: Handle 11.0 vs 10.6 style comparisons, and other complexities

    # Check format
    if binary_format not in _mac_binary_formats(version, arch):
        return False

    return True

[brettcannon]

The other "fun" part of tags is they were designed to be opaque strings, so parsing them was never part of the design. And I personally wouldn't want to try and make that be forward/backwards-compatible as I feel like that's asking for trouble. I also bet generating the tag combinations isn't the expensive part, but the parts that query the OS which you have to do anyway to validate compatibility. If we were dealing with list lengths in the millions I would understand, but we are dealing with maybe a thousand at worst, and we have platform_tags() if you want to control which part of the tag triple you want to look at.

If people want a more ergonomic, high-level I'm happy to talk about that, but I don't want to get sucked into premature optimization below that API surface when I don't know how much computation we can actually avoid in the common case. Or put another way, any(tag == incoming for incoming in mac_platforms()) is already lazy thanks to mac_platforms() being a generator, so how much are you really shaving off in CPU costs by parsing the tag itself?

[rgommers]

For me it has nothing to do with cost of computation, that seems basically irrelevant for any packaging tools that create, install, or modify wheels. And I don't think anyone suggested otherwise? The API is cumbersome, not slow.

[brettcannon]

The API is cumbersome, not slow.

I've opened https://github.com/pypa/packaging/issues/602 so we can discuss what sort of API people may want.