Document Threads and Forking (through ProcessPool and friends)

[Goobley]

If ctx.Nthreads > 1 on fork, then only main thread is copied, and none of the workers are cloned (as per the definition of fork(2)). If Nthreads is set to one and then increased in each child process then all is well. This happens with a Context in global state on creating a ProcessPoolExecutor.

Interestingly if ctx.Nthreads is left > 1, everything still runs (albeit only 1 thread per process). This is due to the way the task library works. It spawns N-1 threads, with the Nth being the main thread. When we call scheduler_join in the main thread, it jumps in and does the work (as if the other workers were busy).

This is all surprisingly close to "as one would/should expect" IMO, however it should be documented somewhere. For the most part if a job can use inter-process concurrency, it is unlikely to need the sub-process multi-threaded formal solutions that ctx.Nthreads exists for, so in most places ctx.Nthreads should be set to 1 before spawning. If multiple threads are needed in the children then there are two simple options off the top of my head: set ctx.Nthreads > 1 and the scheduler will happily spin up threads, or don't rely on the global state fork behaviour, and pass the context through the initialiser options available in ProcessPoolExecutor.