pyo3 icon indicating copy to clipboard operation
pyo3 copied to clipboard

Expose unsafe wrappers for Py_BEGIN_CRITICAL_SECTION_MUTEX API

Open ngoldbaum opened this issue 1 month ago • 4 comments

Python 3.14 introduced new variants for the critical section macros that accept a PyMutex instead of a PyObject. See https://github.com/python/cpython/issues/133296, https://github.com/capi-workgroup/decisions/issues/67, and https://github.com/python/cpython/pull/135899 for more details.

I also was responsible for making sure this landed in 3.14, so shipping this PR is special to me as it completes the loop :)

So far I don't think there are any open source uses yet because it's so new and it's 3.14-specific.

The new APIs accept an EnteredCriticalSection which exposes unsafe get and get_mut methods because the caller needs to guarantee that the closure doesn't unsafely release the resource protected by the PyMutex.

I also expanded the docs for the existing critical section API wrappers to clarify things a bit.

ngoldbaum avatar Nov 21 '25 20:11 ngoldbaum

Argh, there are deeper issues with this.

I can't actually recover the no-op semantics on the GIL-enabled build. The key difference is that in Rust, the mutexes are wrapper types. We can't access the data stored in the wrappers without acquiring the lock. In C, PyMutex doesn't wrap anything and there's nothing different about access to the protected resource before and after locking the mutex.

That means that making the GIL-enabled version of the critical section a no-op is fine in C, but in Rust we need to do something fancier. I guess I could lock the mutex during the closure on the GIL-enabled build? Or use GILProtected maybe? But ultimately I still need to lock the mutex, otherwise I hit deadlocks....

This isn't a problem with the regular critical section APIs because from the point of view of PyO3 taking a critical section on an object doesn't unlock a wrapped resource.

It's a shame because this all works fine on the free-threaded build. Hmm...

ngoldbaum avatar Nov 21 '25 21:11 ngoldbaum

I marked this ready for review but I expect it to have at least one more round. I left a couple questions for others and comments for myself inline.

ngoldbaum avatar Nov 24 '25 20:11 ngoldbaum

I think the last push satisfies all your comments and concerns.

ngoldbaum avatar Dec 04 '25 00:12 ngoldbaum

I think tests are green now. I also added a release note for the moved functions.

ngoldbaum avatar Dec 04 '25 18:12 ngoldbaum