spawn-multiprocessing and frozen mcsolve

[goerz]

Different platforms have some subtle differences in how multiprocessing works: On linux, processes "fork". My basic understanding is that sub-processes are created as copies of the parent process, thus inheriting their entire state (global variables/functions). Windows, and macOS with Python >= 3.8 uses "spawn". There, I believe subprocesses start from blank slate, and relevant global state is injected via IPC. Sadly, limitations of the pickle protocol can cause "spawn"-based multiprocessing to have problems.

Jupyter notebooks are affected by this: you cannot use multiprocessing-map using functions defined within the notebook.

Within qutip, this definitely affects the parallel_map function: It currently will cause freezes on Windows (See https://qucontrol.github.io/krotov/v1.0.0/notebooks/08_example_ensemble.html). With Python 3.8, macOS is also affected by this. See https://github.com/qutip/qutip-notebooks/issues/100

One possible workaround is to use a third-party replacement for multiprocessing. An work-in-progress implementation of this is in https://github.com/qutip/qutip/pull/1092, using loky.

In #1197, we also identified a freeze in mcsolve when running tests on macOS/Python 3.8. This is suspected to be an issue with spawn-based multiprocessing, although we haven't been able to determine this with complete confidence.

If the freeze is indeed caused by spawn-multiprocessing, the problem should also show up on Windows. Is anyone running Windows able to reproduce this?

[nonhermitian]

This may help https://github.com/Qiskit/qiskit-terra/pull/3491

[Ericgig]

I can get the same error on linux with:

from qutip.testing import run
import multiprocessing as mp
mp.set_start_method('spawn')
run()

It hangs on qutip/tests/test_mcsolve.py::test_MCTDFunc. So maybe, set_start_method('fork') could probably fix the problem on mac. Also this hang, both at console and jupyter:

import multiprocessing as mp
mp.set_start_method('spawn')

def f(i):
    return i+1

p = mp.Pool()
p.map(f, [1,2,3,4])

[berryboy2012]

Different platforms have some subtle differences in how multiprocessing works: On linux, processes "fork". My basic understanding is that sub-processes are created as copies of the parent process, thus inheriting their entire state (global variables/functions). Windows, and macOS with Python >= 3.8 uses "spawn". There, I believe subprocesses start from blank slate, and relevant global state is injected via IPC. Sadly, limitations of the pickle protocol can cause "spawn"-based multiprocessing to have problems.

Jupyter notebooks are affected by this: you cannot use multiprocessing-map using functions defined within the notebook.

Within qutip, this definitely affects the parallel_map function: It currently will cause freezes on Windows (See https://qucontrol.github.io/krotov/v1.0.0/notebooks/08_example_ensemble.html). With Python 3.8, macOS is also affected by this. See qutip/qutip-notebooks#100

One possible workaround is to use a third-party replacement for multiprocessing. An work-in-progress implementation of this is in #1092, using loky.

In #1197, we also identified a freeze in mcsolve when running tests on macOS/Python 3.8. This is suspected to be an issue with spawn-based multiprocessing, although we haven't been able to determine this with complete confidence.

If the freeze is indeed caused by spawn-multiprocessing, the problem should also show up on Windows. Is anyone running Windows able to reproduce this?

Windows doesn't have 'fork' at all, if we want to make parallel_map work under windows, we have to find ways to 'spawn' efficiently.

[Ericgig]

Version 5 has loky (and mpi) based parallel maps available for when multiprocessing doesn't want to cooperate. All maps are tests of linux, mac and windows.