gh-118789: Add `PyUnstable_Object_ClearWeakRefsExceptCallbacks`

[colesbury]

This exposes _PyWeakref_ClearWeakRefsExceptCallbacks as an unstable C-API function to provide a thread-safe mechanism for clearing weakrefs without executing callbacks.

Some C-API extensions need to clear weakrefs without calling callbacks, such as after running finalizers like we do in subtype_dealloc. Previously they could use _PyWeakref_ClearRef on each weakref, but that's not thread-safe in the free-threaded build.

• Issue: gh-118789

---

📚 Documentation preview 📚: https://cpython-previews--118807.org.readthedocs.build/

[vstinner]

I suggest to rename the function to: PyObject_ClearWeakRefs().

• I don't think that it deserves to belong to the unstable API, since it's stable for many years.
• It doesn't belong to PyWeakref API, since the argument is a generic object.
• I don't think that there is an use case to call the callbacks. If tomorrow, someone comes with such use case, we can add a new function. I don't think that adding a parameter for that would be useful right now.

[colesbury]

I suggest to rename the function to: PyObject_ClearWeakRefs()

We already have PyObject_ClearWeakRefs: it clears the weakrefs and calls callbacks. It's the more commonly used API.

The use case for both functions is that types with user finalizers (i.e., __del__ methods) need to perform the following sequence if they want to be correct:

• Clear weakrefs and call callbacks (i.e., PyObject_ClearWeakRefs)
• Call finalizer (i.e, PyObject_CallFinalizerFromDealloc)
• Clear weakrefs again, but don't call callbacks (i.e., PyUnstable_Weakref_ClearWeakRefsExceptCallbacks)

The third step avoids leaking memory in case the finalizer added new weakrefs. We do this internally in subtype_dealloc, but extensions can't always delegate to subtype_dealloc.

Previously, the third step was implemented as:

      PyWeakReference** list =
          (PyWeakReference**)PyObject_GET_WEAKREFS_LISTPTR(self);
      while (*list)
        _PyWeakref_ClearRef(*list);
    }

But that's not thread-safe without the GIL.

It doesn't belong to PyWeakref API, since the argument is a generic object

I'll adjust the name and add the check you suggest.

[vstinner]

Why do you want to put it in the PyUnstable API?

[colesbury]

Why do you want to put it in the PyUnstable API?

That's what I understood from @Yhg1s's comments on Discord (#c-api) as the appropriate name/stability for 3.13.

I'd certainly prefer something outside of the PyUnstable API as the end state for this (i.e., in 3.14).

[vstinner]

I suggest to only do the revert in 3.13, and add the function only in 3.14.

[colesbury]

If we only add the function in 3.14, then there will be no way for C-API extensions that need this functionality to be made thread-safe in 3.13.

[colesbury]

I'll open a https://github.com/capi-workgroup/decisions issue for 3.14

[colesbury]

@encukou, would you please review this?

[encukou]

Unstable API needs tests; see https://github.com/colesbury/cpython/pull/2. Otherwise, this looks good to me.

@Yhg1s, do you want this in 3.13? As mentioned in the issue, projects that reimplement subtype_dealloc need this to be thread-safe in free-threading builds. The proper solution is (IMO) to give them API so they don't need to reimplement subtype_dealloc, but that's definitely not coming in 3.13.

[miss-islington-app[bot]]

Thanks @colesbury for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13. 🐍🍒⛏🤖

[bedevere-app[bot]]

GH-120695 is a backport of this pull request to the 3.13 branch.

[vstinner]

Congrats @colesbury :-) PyTorch should be happy with this new API if I understood correctly.