mesonbuild
Disable abi3 wheel tag for PyPy
Pip does not support installing PyPy wheels with the abi3 tag. If you build a wheel for PyPy and set the ABI tag to abi3, pip will not be able to install it.
$ docker run --rm -it quay.io/pypa/manylinux2014_x86_64
[root@cd7b2d465170 /]# /opt/python/pp39-pypy39_pp73/bin/python
Python 3.9.19 (a2113ea87262, Apr 21 2024, 05:40:24)
[PyPy 7.3.16 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> from pip._internal.utils.compatibility_tags import get_supported
>>>> for tag in get_supported():
.... print(tag)
....
pp39-pypy39_pp73-manylinux_2_17_x86_64
pp39-pypy39_pp73-manylinux2014_x86_64
Full output:
>>>> from pip._internal.utils.compatibility_tags import get_supported
>>>> for tag in get_supported():
.... print(tag)
....
pp39-pypy39_pp73-manylinux_2_17_x86_64
pp39-pypy39_pp73-manylinux2014_x86_64
pp39-pypy39_pp73-manylinux_2_16_x86_64
pp39-pypy39_pp73-manylinux_2_15_x86_64
pp39-pypy39_pp73-manylinux_2_14_x86_64
pp39-pypy39_pp73-manylinux_2_13_x86_64
pp39-pypy39_pp73-manylinux_2_12_x86_64
pp39-pypy39_pp73-manylinux2010_x86_64
pp39-pypy39_pp73-manylinux_2_11_x86_64
pp39-pypy39_pp73-manylinux_2_10_x86_64
pp39-pypy39_pp73-manylinux_2_9_x86_64
pp39-pypy39_pp73-manylinux_2_8_x86_64
pp39-pypy39_pp73-manylinux_2_7_x86_64
pp39-pypy39_pp73-manylinux_2_6_x86_64
pp39-pypy39_pp73-manylinux_2_5_x86_64
pp39-pypy39_pp73-manylinux1_x86_64
pp39-pypy39_pp73-linux_x86_64
pp39-none-manylinux_2_17_x86_64
pp39-none-manylinux2014_x86_64
pp39-none-manylinux_2_16_x86_64
pp39-none-manylinux_2_15_x86_64
pp39-none-manylinux_2_14_x86_64
pp39-none-manylinux_2_13_x86_64
pp39-none-manylinux_2_12_x86_64
pp39-none-manylinux2010_x86_64
pp39-none-manylinux_2_11_x86_64
pp39-none-manylinux_2_10_x86_64
pp39-none-manylinux_2_9_x86_64
pp39-none-manylinux_2_8_x86_64
pp39-none-manylinux_2_7_x86_64
pp39-none-manylinux_2_6_x86_64
pp39-none-manylinux_2_5_x86_64
pp39-none-manylinux1_x86_64
pp39-none-linux_x86_64
py39-none-manylinux_2_17_x86_64
py39-none-manylinux2014_x86_64
py39-none-manylinux_2_16_x86_64
py39-none-manylinux_2_15_x86_64
py39-none-manylinux_2_14_x86_64
py39-none-manylinux_2_13_x86_64
py39-none-manylinux_2_12_x86_64
py39-none-manylinux2010_x86_64
py39-none-manylinux_2_11_x86_64
py39-none-manylinux_2_10_x86_64
py39-none-manylinux_2_9_x86_64
py39-none-manylinux_2_8_x86_64
py39-none-manylinux_2_7_x86_64
py39-none-manylinux_2_6_x86_64
py39-none-manylinux_2_5_x86_64
py39-none-manylinux1_x86_64
py39-none-linux_x86_64
py3-none-manylinux_2_17_x86_64
py3-none-manylinux2014_x86_64
py3-none-manylinux_2_16_x86_64
py3-none-manylinux_2_15_x86_64
py3-none-manylinux_2_14_x86_64
py3-none-manylinux_2_13_x86_64
py3-none-manylinux_2_12_x86_64
py3-none-manylinux2010_x86_64
py3-none-manylinux_2_11_x86_64
py3-none-manylinux_2_10_x86_64
py3-none-manylinux_2_9_x86_64
py3-none-manylinux_2_8_x86_64
py3-none-manylinux_2_7_x86_64
py3-none-manylinux_2_6_x86_64
py3-none-manylinux_2_5_x86_64
py3-none-manylinux1_x86_64
py3-none-linux_x86_64
py38-none-manylinux_2_17_x86_64
py38-none-manylinux2014_x86_64
py38-none-manylinux_2_16_x86_64
py38-none-manylinux_2_15_x86_64
py38-none-manylinux_2_14_x86_64
py38-none-manylinux_2_13_x86_64
py38-none-manylinux_2_12_x86_64
py38-none-manylinux2010_x86_64
py38-none-manylinux_2_11_x86_64
py38-none-manylinux_2_10_x86_64
py38-none-manylinux_2_9_x86_64
py38-none-manylinux_2_8_x86_64
py38-none-manylinux_2_7_x86_64
py38-none-manylinux_2_6_x86_64
py38-none-manylinux_2_5_x86_64
py38-none-manylinux1_x86_64
py38-none-linux_x86_64
py37-none-manylinux_2_17_x86_64
py37-none-manylinux2014_x86_64
py37-none-manylinux_2_16_x86_64
py37-none-manylinux_2_15_x86_64
py37-none-manylinux_2_14_x86_64
py37-none-manylinux_2_13_x86_64
py37-none-manylinux_2_12_x86_64
py37-none-manylinux2010_x86_64
py37-none-manylinux_2_11_x86_64
py37-none-manylinux_2_10_x86_64
py37-none-manylinux_2_9_x86_64
py37-none-manylinux_2_8_x86_64
py37-none-manylinux_2_7_x86_64
py37-none-manylinux_2_6_x86_64
py37-none-manylinux_2_5_x86_64
py37-none-manylinux1_x86_64
py37-none-linux_x86_64
py36-none-manylinux_2_17_x86_64
py36-none-manylinux2014_x86_64
py36-none-manylinux_2_16_x86_64
py36-none-manylinux_2_15_x86_64
py36-none-manylinux_2_14_x86_64
py36-none-manylinux_2_13_x86_64
py36-none-manylinux_2_12_x86_64
py36-none-manylinux2010_x86_64
py36-none-manylinux_2_11_x86_64
py36-none-manylinux_2_10_x86_64
py36-none-manylinux_2_9_x86_64
py36-none-manylinux_2_8_x86_64
py36-none-manylinux_2_7_x86_64
py36-none-manylinux_2_6_x86_64
py36-none-manylinux_2_5_x86_64
py36-none-manylinux1_x86_64
py36-none-linux_x86_64
py35-none-manylinux_2_17_x86_64
py35-none-manylinux2014_x86_64
py35-none-manylinux_2_16_x86_64
py35-none-manylinux_2_15_x86_64
py35-none-manylinux_2_14_x86_64
py35-none-manylinux_2_13_x86_64
py35-none-manylinux_2_12_x86_64
py35-none-manylinux2010_x86_64
py35-none-manylinux_2_11_x86_64
py35-none-manylinux_2_10_x86_64
py35-none-manylinux_2_9_x86_64
py35-none-manylinux_2_8_x86_64
py35-none-manylinux_2_7_x86_64
py35-none-manylinux_2_6_x86_64
py35-none-manylinux_2_5_x86_64
py35-none-manylinux1_x86_64
py35-none-linux_x86_64
py34-none-manylinux_2_17_x86_64
py34-none-manylinux2014_x86_64
py34-none-manylinux_2_16_x86_64
py34-none-manylinux_2_15_x86_64
py34-none-manylinux_2_14_x86_64
py34-none-manylinux_2_13_x86_64
py34-none-manylinux_2_12_x86_64
py34-none-manylinux2010_x86_64
py34-none-manylinux_2_11_x86_64
py34-none-manylinux_2_10_x86_64
py34-none-manylinux_2_9_x86_64
py34-none-manylinux_2_8_x86_64
py34-none-manylinux_2_7_x86_64
py34-none-manylinux_2_6_x86_64
py34-none-manylinux_2_5_x86_64
py34-none-manylinux1_x86_64
py34-none-linux_x86_64
py33-none-manylinux_2_17_x86_64
py33-none-manylinux2014_x86_64
py33-none-manylinux_2_16_x86_64
py33-none-manylinux_2_15_x86_64
py33-none-manylinux_2_14_x86_64
py33-none-manylinux_2_13_x86_64
py33-none-manylinux_2_12_x86_64
py33-none-manylinux2010_x86_64
py33-none-manylinux_2_11_x86_64
py33-none-manylinux_2_10_x86_64
py33-none-manylinux_2_9_x86_64
py33-none-manylinux_2_8_x86_64
py33-none-manylinux_2_7_x86_64
py33-none-manylinux_2_6_x86_64
py33-none-manylinux_2_5_x86_64
py33-none-manylinux1_x86_64
py33-none-linux_x86_64
py32-none-manylinux_2_17_x86_64
py32-none-manylinux2014_x86_64
py32-none-manylinux_2_16_x86_64
py32-none-manylinux_2_15_x86_64
py32-none-manylinux_2_14_x86_64
py32-none-manylinux_2_13_x86_64
py32-none-manylinux_2_12_x86_64
py32-none-manylinux2010_x86_64
py32-none-manylinux_2_11_x86_64
py32-none-manylinux_2_10_x86_64
py32-none-manylinux_2_9_x86_64
py32-none-manylinux_2_8_x86_64
py32-none-manylinux_2_7_x86_64
py32-none-manylinux_2_6_x86_64
py32-none-manylinux_2_5_x86_64
py32-none-manylinux1_x86_64
py32-none-linux_x86_64
py31-none-manylinux_2_17_x86_64
py31-none-manylinux2014_x86_64
py31-none-manylinux_2_16_x86_64
py31-none-manylinux_2_15_x86_64
py31-none-manylinux_2_14_x86_64
py31-none-manylinux_2_13_x86_64
py31-none-manylinux_2_12_x86_64
py31-none-manylinux2010_x86_64
py31-none-manylinux_2_11_x86_64
py31-none-manylinux_2_10_x86_64
py31-none-manylinux_2_9_x86_64
py31-none-manylinux_2_8_x86_64
py31-none-manylinux_2_7_x86_64
py31-none-manylinux_2_6_x86_64
py31-none-manylinux_2_5_x86_64
py31-none-manylinux1_x86_64
py31-none-linux_x86_64
py30-none-manylinux_2_17_x86_64
py30-none-manylinux2014_x86_64
py30-none-manylinux_2_16_x86_64
py30-none-manylinux_2_15_x86_64
py30-none-manylinux_2_14_x86_64
py30-none-manylinux_2_13_x86_64
py30-none-manylinux_2_12_x86_64
py30-none-manylinux2010_x86_64
py30-none-manylinux_2_11_x86_64
py30-none-manylinux_2_10_x86_64
py30-none-manylinux_2_9_x86_64
py30-none-manylinux_2_8_x86_64
py30-none-manylinux_2_7_x86_64
py30-none-manylinux_2_6_x86_64
py30-none-manylinux_2_5_x86_64
py30-none-manylinux1_x86_64
py30-none-linux_x86_64
pp39-none-any
py39-none-any
py3-none-any
py38-none-any
py37-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any
Thanks for working on this @lpsinger. The current behavior certainly doesn't seem quite right - we should never be producing something with a tag that pip won't understand.
Related to gh-624. If I understand correctly, with the build option discussed in gh-624 you can disable asking for limited API usage, which will then produce a tag like pp310-pypy310_pp73-linux_x86_64 that can be installed. This PR is a little different in that you want to avoid having to think about using that build option at all, still use the limited C API, but produce a tag like pp310-pypy310_pp73-linux_x86_64. Is that right? If so, that seems reasonable to me, since it reduced friction and the end result should be the same as erroring out and forcing the user to specify -C-Dpython.allow_limited_api=false.
CI is a little unhappy, the tests need updating to match:
FAILED tests/test_tags.py::test_tag_stable_abi - AssertionError: assert 'pp310-pypy310_pp73-linux_x86_64' == 'pp310-abi3-linux_x86_64'
- pp310-abi3-linux_x86_64
+ pp310-pypy310_pp73-linux_x86_64
FAILED tests/test_wheel.py::test_limited_api - AssertionError: assert 'pypy310_pp73' == 'abi3'
This PR is a little different in that you want to avoid having to think about using that build option at all, still use the limited C API, but produce a tag like
pp310-pypy310_pp73-linux_x86_64. Is that right?
That's correct. I'm looking to avoid having to add this line to projects that use the limited API and cibuildwheel: https://github.com/nasa-gcn/hpx/blob/main/pyproject.toml#L81-L84
CI is a little unhappy, the tests need updating to match:
I pushed a change for that; let's see if it fixes the CI.
My understanding is that PyPy supports the limited API but does not use a dedicated filename suffix for extension modules compiled for the limited API. The Python support in Meson implies the same understanding https://github.com/mesonbuild/meson/blob/cfd57180eef9036c7167c5682b9f3055a540fccc/mesonbuild/scripts/python_info.py#L103-L107
What I am not sure about is whether, when using the limited API, PyPy implements a stable ABI, and thus the resulting wheels should be taggeg abi3 or not. If this is the case, the problem being solved here is a bug in pip and not something we need to work-around in meson-python.
@mattip Can you confirm that my understanding is correct?
I added the tool.meson-python.limited-api setting based on the assumption that PyPy supports the limited API without a dedicated module filename suffix and thus it is not sufficient to look at the filenames of the installed extension modules to determine whether a package uses the limited API only. If the assumption is wrong and PyPy does not support the limited API or supports it without exporting a stable ABI, the setting could in principle be removed.
PyPy does not implement a stable ABI. Wheels for PyPy should not be tagged abi3. The output of pip's get_supported is correct.
Thanks for the clarification @mattip. My reasoning for implementing the tool.meson-python.limited-api setting is wrong. I think we should remove this setting and rely on the filename suffix of the installed extension modules to decide the ABI tag for the created wheel. We would still accept the parameter for backward compatibility but emit a deprecation warning when we encounter it. @rgommers, what do you think?
One of the reasons to remove the tool.meson-python.limited-api setting is that it is misnamed. When I introduced it, I thought that using the limited API always results in a stable ABI, and thus in wheels to be tagged with the abi3 tag. However, this is not the case on PyPy. Therefore, mixing the two concepts is not a good idea. To my defense, the CPython documentation does not make this distinction at all.
You can’t do that on windows. No suffix there.
In scikit-build-core, I use wheel.py-api, which is the wheel tag and you can get both limited api and pythonless wheels from that, see https://scikit-build-core.readthedocs.io/en/latest/configuration.html#customizing-the-output-wheel.
Setting something like cp311 is ignored on CPython < 3.11, PyPy, and free-threaded Python.
IIRC, on windows there is no suffix specific to abi3 extension modules but there is a Python ABI specific suffix and the general .pyd suffix. I was thinking of mapping the .pyd suffix to the abi3 wheel tag. This is not fool proof, but I don't expect it to be a problem in practice. I need to go back and check my reasoning from when I implemented the tool.meson-python.limited-api setting.
@henryiii Thanks for the pointer to how scikit-build-core solves this. However, it is not clear to me how you handle the case when the user requires the stable ABI tag abi3 but the package is built for PyPy. Do you simply override the user choice in this case?
My impression is that the current build option is fine, since it works as designed for CPython. For PyPy, proceeding with the regular minor-version-specific tag as done in this PR would seem reasonable to me. The question I have is whether we should be emitting a warning when doing that.
However, this is not the case on PyPy. Therefore, mixing the two concepts is not a good idea.
PyPy could implement it in the future in principle (it's just a missing feature now), so I wouldn't attach too much value to this since conceptually it does make sense.
Moreover, PyPy is probably going to fade away pretty quickly over the next 1-2 years - there is unlikely to be a 3.11 release for PyPy, so next year when Python 3.10 starts being dropped that's the end of it. Conda-forge is also starting to drop PyPy builds because of maintenance issues and no real future. Hence deprecating because of PyPy isn't needed imho.
The way selecting cp311 works is that it's only applicable on CPython with GIL 3.11+. Any older version or other interpreter (including PyPy) ignores this choice. (It wasn't quite right for PyPy until the most recent release, when I was working on free-threading I cleaned it up a little). A user is expected to skip requesting it CMake like this:
if(NOT "${SKBUILD_SABI_COMPONENT}" STREQUAL "")
python_add_library(some_ext MODULE WITH_SOABI USE_SABI 3.11 ...)
else()
python_add_library(some_ext MODULE WITH_SOABI ...)
endif()
(USE_SABI also sets the Limited API define) Though I haven't checked to see exactly how CMake's FindPython module handles it, it might do the right thing (IMO which is to ignore this setting if it doesn't apply).
Not setting the define just means you are not limited to the Limited API, so shouldn't hurt user code.
Moreover, PyPy is probably going to fade away pretty quickly over the next 1-2 years - there is unlikely to be a 3.11 release for PyPy, so next year when Python 3.10 starts being dropped that's the end of it. Conda-forge is also starting to drop PyPy builds because of maintenance issues and no real future. Hence deprecating because of PyPy isn't needed imho.
Does that mean that this is a "won't fix?"
Does that mean that this is a "won't fix?"
I don't think so. This is a minor change that to me seems strictly an improvement over the status quo. So I'd be in favor of merging it as is.
The larger change @dnicolodi was thinking about (this comment above) is orthogonal, and may or may not happen. I'd suggest that it shouldn't be blocking for this PR.