pyo3 icon indicating copy to clipboard operation
pyo3 copied to clipboard

WIP: declare free-threaded support in pymodule macro

Open ngoldbaum opened this issue 1 year ago • 1 comments

This is a WIP attempt to add a way to declare free-threaded support in the pymodule macro. I'm going on a trip soon so I'm opening this just in case someone else needs to finish it up before I get back.

Some wrinkles:

  • Right now I'm only allowing one global setting per pymodule. In principle you could add submodules that don't support free-threaded python to a parent module that does, but for now I think we can ignore that. Unfortunately it's tricky to thread the extra argument through down into where the actual FFI calls happen.
  • Not sure how to make modules added via add_wrapped properly inherit free-threaded support from the parent module for similar reasons.
  • How do we want to handle testing this?
    • Should I mark all the existing modules in the pytests as supporting free-threading?
      • If I don't do this, then under the free-threaded build the pytests will re-enable the GIL at runtime. Unfortunately it's an all-or-nothing thing.
      • I can still test this PR by defining a module that says it needs the GIL, and then import it in a separate process so the runtime re-enabling doesn't "poison" the parent process.
      • I added some code to the NumPy tests to make sure the GIL doesn't get re-enabled at runtime when new modules are added. I think the PyO3 pytests could probably just use that code.
    • Alternatively, I could run the tests with PYTHON_GIL=0, which means the GIL is never re-enabled at runtime no matter what metadata is attached to the module.
      • If I do it this way, there's no way to test that this PR actually does anything.

ngoldbaum avatar Oct 01 '24 22:10 ngoldbaum

Just a note that this should probably only be merged once all the "RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module foo" warnings in the python tests are fixed in the free-threaded build.

ngoldbaum avatar Oct 19 '24 23:10 ngoldbaum

Just a note that this should probably only be merged once all the "RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module foo" warnings in the python tests are fixed in the free-threaded build.

This is done now and I think all the tests are passing so I'm going to move this out of draft phase and update the PR description.

ngoldbaum avatar Oct 21 '24 23:10 ngoldbaum

This probably needs some discussion in the new guide section on free-threading support.

ngoldbaum avatar Oct 21 '24 23:10 ngoldbaum

woohoo! looks like tests are passing except for the codecov issue affecting all PRs right now.

ngoldbaum avatar Oct 23 '24 16:10 ngoldbaum

It occurred to me while working on something else that this PR should remove the UNSAFE_PYO3_BUILD_FREE_THREADED uses in the CI scripts a well.

EDIT: done

ngoldbaum avatar Oct 31 '24 15:10 ngoldbaum