py_cc_toolchain should contain EXT_SUFFIX

[jhance]

See https://github.com/bazelbuild/bazel-central-registry/pull/4539#discussion_r2080143661 for some context.

This value is needed to create python extensions. Note that there are a few different names that can be used, for example .so, .abi3.so, and .cpython-312-x86_64-linux-gnu.so. The latter is preferable when not using the python limited api because it prevents runfiles conflicts between python versions properly, meaning that transitions can work without completely breaking things.

I believe for autoconfigured pythons, you could obtain this value like so:

❯ python -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))'
.cpython-312-x86_64-linux-gnu.so

[rickeylev]

These peps look relevant:

• https://peps.python.org/pep-3149/
• https://peps.python.org/pep-0384/

IIUC:

• .so -- legacy value. Basically equivalent to abi3.so today, I think.
• abi3.so -- means it's using the limited API and following the stable ABI. Can be used by any 3.x python.
• .so, e.g. cpython-312-x86_64-linux-gnu.so -- unstable API/ABI

Macs and windows also have multiple suffixes, I think? Macs: .dylib, .so. Windows: .dll, .pyd. I dunno where those fit in or get reported by, though.

In any case, for now, +1 to adding PyCcToolchainInfo.ext_suffix -- this field can directly reflect what sysconfig's EXT_SUFFIX reports. Is there, perhaps, a sysconfig var for ".abi3" ?

avoids runfiles conflicts

i.e. if you have multiple binaries using different python versions that get merged into the same runfiles tree? Yeah, good point. (Sounds like you went through some python 2 to 3 work in bazel 😅). If an extension module produces an abi3.so file, then it should be usable in multiple python versions, so, in theory, a runfiles conflict shouldn't occur (in practice, even if the same output is produced, this seems prone to other config differences causing an action conflict, so i'm not sure how possible this is in practice).

fyi to @nicholasjng and @EthanSteinberg who would probably be interested in this as well

[jhance]

I can't find any indication of an abi3 sysconfig var. But it should be the same on all platforms for a give python version so I suspect that's fine.

I feel like the way I would expect the limited API to work with bazel would be for it to be an option you can enable on your toolchain (setting the value of it to the floor you want) and then the cc extension rule ought to inject the -D copts. Then you would use that when generating your public package or whatever.

And yes, our repository has support for mixed versions in one runfiles trees, although we use it a lot less now that python2 is eradicated. It can be pretty convenient for if one thing has an issue in an upgrade (usually the runfiles conflicts would come from integration tests)

[aignas]

I agree that the py_cc_toolchain should have the unstable ABI ext suffix, but I am not sure that the toolchain should select whether abi3.so or the unstable ABI is used.

I thought that whether abi3.so is used is a property of the users code and not py_cc_toolchain. What one needs to do to enable it is to do as described in the docs.

[nicholasjng]

Thanks for the tag @rickeylev. There is also a route via importlib.machinery.EXTENSION_SUFFIXES [1], which seems to be a good one (contained in the standard library, available from Python 3.3+):

>>> import importlib.machinery
>>> importlib.machinery.EXTENSION_SUFFIXES
['.cpython-313-darwin.so', '.abi3.so', '.so']

I received an issue on nanobind-bazel for this (https://github.com/nicholasjng/nanobind-bazel/issues/61), and the (in my opinion compelling) argument is that with these versioned ext suffixes, it's possible to create minor-agnostic wheels with C++ extensions, because Python's dlopen() can select the right one for the current interpreter from a list of built extensions for different Python versions.

If I understand correctly, extensions are built against ABI3 if and only if Py_LIMITED_API is set, so I would agree with the above:

I feel like the way I would expect the limited API to work with bazel would be for it to be an option you can enable on your toolchain (setting the value of it to the floor you want) and then the cc extension rule ought to inject the -D copts. Then you would use that when generating your public package or whatever.

NB: Windows does not seem to add .abi3 to the extension tag when building against the stable ABI, so for a fully compliant implementation, the target platform might have to be accessed as well.

Sources:

• [1] https://discuss.python.org/t/is-there-a-modern-use-case-for-backwards-compatible-so-extension-suffix/58544