multiprocessing-logging
multiprocessing-logging copied to clipboard
Add support for `spawn` and `forkserver` start methods
Thanks for the library! As of Python 3.8, the default start method for multiprocessing processes on macOS is spawn
; see here.
Unfortunately, as noted in #40 and #28, the current implementation of multiprocessing-logging
does not support the spawn
start method. It seems forkserver
has the same type of issue.
Suggestion is to add support for spawn
and forkserver
methods, or update documentation to note that the fork
start method should be used.
Example to reproduce:
import logging
import multiprocessing
import time
from multiprocessing_logging import install_mp_handler
logging.basicConfig(
filename="output.txt",
format="%(asctime)s: %(message)s",
filemode="w",
level=logging.INFO,
)
install_mp_handler()
def worker(x):
logging.info(f"sleeping {x} seconds")
time.sleep(x)
if __name__ == "__main__":
logging.info("start")
# https://docs.python.org/3/library/multiprocessing.html#multiprocessing.get_context
ctx = multiprocessing.get_context("spawn")
with ctx.Pool(4) as p:
p.map(worker, [1, 2, 3, 4])
logging.info("stop")
spawn
or forkserver
start methods result in the following output:
2020-10-14 23:05:36,979: sleepi2020-10-14 23:05:41,030: stop
fork
start method results in the following output:
2020-10-14 23:09:59,094: start
2020-10-14 23:09:59,103: sleeping 1 seconds
2020-10-14 23:09:59,103: sleeping 2 seconds
2020-10-14 23:09:59,104: sleeping 3 seconds
2020-10-14 23:09:59,105: sleeping 4 seconds
2020-10-14 23:10:03,169: stop
There's a PR that may allow other startmethods but I haven't reviewed it. I'll update the documentation for a quick fix.
The approach of this module does not allow for other start methods. Please see #49 for more details.