gh-71936: Fix race condition in multiprocessing.Pool

[hattya]

trafficstars

Proxes of shared objects register a Finalizer in BaseProxy._incref(), and it will call BaseProxy._decref() when it is GCed. This may cause a race condition with Pool(maxtasksperchild=None) on Windows.

A connection will be closed and raised TypeError when a GC occurs between _ConnectionBase._check_writable() and _ConnectionBase._send_bytes() in _ConnectionBase.send() in the second or later task.

Finalizers should be processed after each task is finished, because all proxies of shared objects will lost their references in the next task.

Also, BaseProxy does not count references well.

• Issue: gh-71936

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[cpython-cla-bot[bot]]

All commit authors signed the Contributor License Agreement.

[hattya]

Steps to reproduce:

> xcopy /I "%ProgramFiles%\Python310\Lib\multiprocessing" multiprocessing
> type a.patch
--- a/multiprocessing/connection.py
+++ b/multiprocessing/connection.py
@@ -282,6 +282,8 @@
             _CloseHandle(self._handle)

         def _send_bytes(self, buf):
+            if (10, 1) in util._finalizer_registry:
+                import gc; gc.collect()
             ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
             try:
                 if err == _winapi.ERROR_IO_PENDING:
> patch -p1 < a.patch
> type a.py
#!/usr/bin/env python
import multiprocessing as mp

def foo(lock, i):
    def func(x):
        # for circular reference
        with lock:
            pass
        if x > 0:
            return func(x - 1)

    func(i)

if __name__ == '__main__':
    m = mp.Manager()
    lock = m.Lock()
    with mp.Pool(2) as pool:
        for _ in pool.starmap(foo, ((lock, i) for i in range(10))):
            pass
> a.py

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[basnijholt]

Like I mentioned here https://github.com/python/cpython/issues/71936#issuecomment-2164009639, this patch resolves the problem I am running into on Python 3.12.3.

~~@hattya, could you add a news entry like the bot suggests?~~ I see you already did that.