Add freethreading_compatible directive to set Py_mod_gil slot

[lysnikolaou]

Closes #6239.

The reason this is still a draft is that the test should only run under the free-threaded build. Is that something we can do easily? I saw the mechanism that includes limited api tests, but that seems more complex.

[lysnikolaou]

Thanks for the quick response on this @da-woods! I've addressed the feedback.

[da-woods]

I'll leave this open for a little bit so people can bikeshed names. But I think it's basically good

[colesbury]

It'd be great to see this get merged soon as it's a prerequisite for building free-threading compatible wheels for many of the projects that depend on Cython.

[da-woods]

@lysnikolaou and I had a brief chat about this. I think we'd both like to get this merged with the current behaviour in the fairly near future. (From my point of view that means "probably sometime after EuroPython")

My one suggestion was to explicitly document the option as "experimental" - I think we do want to give ourselves freedom to evolve his we treat free-threading for at least as long as it's experimental in CPython.

[scoder]

I'm fine with merging this, but yes, please document it as experimental. It's not obvious that there will be changes later, but it's going to take a while until we understand the user experience well enough to give good answers to it.

One comment, there is no need to declare the directive type since that's handled automatically for a bool default value.

[ngoldbaum]

I tried to compile _pcg64.pyx in numpy with this directive inserted at the top of the file and hit this compilation error when building numpy:

numpy/random/_pcg64.cpython-313t-darwin.so.p/numpy/random/_pcg64.pyx.c:9310:3: error: expected '}'
  {0, NULL}
  ^
numpy/random/_pcg64.cpython-313t-darwin.so.p/numpy/random/_pcg64.pyx.c:9304:51: note: to match this '{'
static PyModuleDef_Slot __pyx_moduledef_slots[] = {
                                                  ^
1 error generated.

Haven't looked further at why this is happening yet.

[ngoldbaum]

Hmm, that ended up being because I had a typo in the directive name, frethreading_compatible=True. Not sure why the miscompilation happened with the typo given the code change in this PR.

[lysnikolaou]

Thanks a lot everyone for the feedback. Pushed a new commit that:

• Fixes the bug that @ngoldbaum discovered (syntax error in single-phase init)
• Adds tests for this under single-phase init
• Mentions in the documentation that this is still experimental
• Removes the directive type declaration like @scoder suggested

[da-woods]

Thanks

[scoder]

The new test is failing in the freethreading CPython build:

Errors from shard 6:
======================================================================
FAIL: runTest (__main__.EndToEndTest.runTest)
[6] End-to-end freethreading_compatible
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cython/cython/runtests.py", line 2040, in runTest
    self.assertEqual(0, res, "non-zero exit status, last output was:\n%r\n-- stdout:%s\n-- stderr:%s\n" % (
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ' '.join(command), out[-1], err[-1]))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 0 != 1 : non-zero exit status, last output was:
'/home/runner/venv-3.13/bin/python -c import default; default.test()'
-- stdout:
-- stderr:Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import default; default.test()
                    ~~~~~~~~~~~~^^
  File "default.py", line 7, in default.test
    assert sys._is_gil_enabled()
AssertionError

https://github.com/cython/cython/actions/runs/9903272103/job/27358534710

[da-woods]

default.py should be assert not sys._is_gil_enabled() instead I think. So it's the test that's wrong rather than the behaviour.

[lysnikolaou]

Hmm, that's weird. The test succeeds locally on my macOS.

default.py should be assert not sys._is_gil_enabled() instead I think. So it's the test that's wrong rather than the behaviour.

I don't think that's correct. By default, the directive is off, which means that the GIL should be enabled after importing the module.

[lysnikolaou]

Oh, it's because the CI job sets PYTHON_GIL=0 so that GIL is not enabled, even if a non-compatible modules is imported. Not sure what the correct fix is here, since we probably don't want to enable the GIL while running the tests.