cython icon indicating copy to clipboard operation
cython copied to clipboard

[ENH] Add a way to set Py_mod_gil slot for modules defined in cython

Open ngoldbaum opened this issue 1 year ago • 6 comments

The free-threaded python build requires that modules explicitly declare whether or not they support running without the GIL using either PyUnstable_Module_SetGIL for single-phase init or the Py_mod_gil slot for multi-phase init.

If the slot isn't set, python warns at runtime about this and re-enables the GIL, unless you force the interpreter to disable the GIL:

In [1]: from numpy import *
<frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to 
load module 'numpy.random.bit_generator', which has not declared that it can run safely without the GIL. 
To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

I tried looking at doing that from inside the definition of a cython module in the cython language, and I came to the conclusion that cython needs to handle this in the compiler, since the changes we need to do to the module need to be in the module definition in C or C++.

I don't think cython can just automatically say that modules rely on the GIL or not, since code may implicitly be locking around the GIL. It makes sense to me to let users explicitly opt-in with a global or module-local compiler directive.

ngoldbaum avatar Jun 11 '24 21:06 ngoldbaum

That makes sense to me.

da-woods avatar Jun 12 '24 07:06 da-woods

I can get started on implementing this.

lysnikolaou avatar Jun 12 '24 07:06 lysnikolaou

I assume this would need both a source-level directive and a front-end flag for cython/cythonize?

For NumPy and SciPy I'd expect we would want to use it as cython --free-threaded-safe (or whatever the flag ends up being called).

rgommers avatar Jun 12 '24 09:06 rgommers

Souce-level directives are automatically command line arguments to the cython command via -X so I don't think anything special needs doing there. (I can't remember about cythonize off the top of my head).

I'm not quite sure why you'd want to use the command-line argument as most of the time though? It seems more like something the source code should know rather than the the build system. (It would be a no-op for non-free-threaded Python builds so wouldn't break anything to set it universally once you'd decided that an individual file was good).

da-woods avatar Jun 12 '24 11:06 da-woods

Ah yes, good point about -X. Nice and easy then:)

CLI arguments: because (a) easier to audit since we have way more .pyx files than build config files, and (b) a cleaner end state (a single flag somewhere, right next to -3) rather than an annotation in every single source file. The source file annotations are also easy to forget when a contributor adds a new .pyx file.

rgommers avatar Jun 12 '24 12:06 rgommers

Minor logistical request: could this issue and the linked PR please be labelled with nogil CPython?

rgommers avatar Jun 23 '24 09:06 rgommers