ericsnowcurrently
A guide for PEP384 and static free C Extensions
One of the big blockers for PEP554 and different Python VM implementations is the use of module state in static globals. Getting rid of this would mean the full migration of all C Extension to PEP384 as well as the removal of any static state. However, many core-devs still have lots of questions on how to correctly do this.
I have a step by step guide on how to migrate a C Extension with statics to a C Extension that uses heap types and no static state. This is what we used to land all recent the PEP384 changes at the core sprint.
I'd like to share this with the community to:
- Make it easier for other core-devs to understand the direction and ease up reviewing.
- Make it easier for other people to work on this (including third-party library owners).
Where would be the right place to share this guide? capi-sig, python-dev, the python docs?
Also, would it make sense to update all the docs in https://docs.python.org/3/extending/extending.html to only use heap types and disallow the use of statics?
I don't think that we need to fully migrate to the stable ABI (PEP 384) to support multiple subinterpreters (PEP 554). They are two different things. I am trying to keep both in mind when designing new API (like multi-phase init, PEP 489, and module state access, PEP 573). My goal is that with new API you get subinterpreter support easily, if not "for free": it should be easy to do the right thing. But we are not there yet. (Never mind that competing with statics on ease of use is a losing race.)
I don't think we should contort the the stdlib with workarounds just to support the stable ABI. We could first add elegant stable ABI, and then use that -- but the stdlib itself doesn't need stable ABI. Supporting subinterpreters, on the other hand, is a good goal in itself. And supporting subinterpreters usually (but not always!) means getting rid of statics.
There is another reason to use the stable ABI: it's easier to implement and use in other implementations of the C-API. That's not something that aligns directly with CPython, though. I'm happy to take patches that migrate to stable ABI and improve the code (e.g. by better readability or adding interpreter support). But if the conversion would ruin performance or make the code harder to read, I'd rather take the patch only as feedback on where the stable ABI is lacking.
To answer your question, I think this repo would be the right place to put the guide. Link to it on capi-sig, and maybe later on python-dev (almost everyone of python-dev who cares reads capi-sig). bpo-26515 tracks adding similar docs to the guide.
Also, would it make sense to update all the docs in https://docs.python.org/3/extending/extending.html to only use heap types and disallow the use of statics?
I can't comment on the other stuff, but updating docs in a way which discourages use of statics is definitely a needed step.
Extension authors would definitely benefit from more documentation on best practices for interacting with the C-API. A dedicated page in the official Python docs on how to take advantage of the stable ABI would be great. Sharing it first with the capi-sig would be a good idea, though for feedback it might be better to surface the doc as some sort of pull request on which people can leave comments (doesn't have to be a PR against the cpython repo). Then in your post you could reference that PR.
This repo is more about tracking tasks and gathering project-oriented info, and I expect it will go away (or fade away) once the project is done. So I'd recommend against putting anything user-facing like that here.
Regardless, discouraging statics for non-const data (including static types) in the existing docs would be a valuable change.
