multiprocessing-logging
multiprocessing-logging copied to clipboard
Add support for `spawn` and `forkserver` start methods
Thanks for the library! As of Python 3.8, the default start method for multiprocessing processes on macOS is spawn; see here.
Unfortunately, as noted in #40 and #28, the current implementation of multiprocessing-logging does not support the spawn start method. It seems forkserver has the same type of issue.
Suggestion is to add support for spawn and forkserver methods, or update documentation to note that the fork start method should be used.
Example to reproduce:
import logging
import multiprocessing
import time
from multiprocessing_logging import install_mp_handler
logging.basicConfig(
filename="output.txt",
format="%(asctime)s: %(message)s",
filemode="w",
level=logging.INFO,
)
install_mp_handler()
def worker(x):
logging.info(f"sleeping {x} seconds")
time.sleep(x)
if __name__ == "__main__":
logging.info("start")
# https://docs.python.org/3/library/multiprocessing.html#multiprocessing.get_context
ctx = multiprocessing.get_context("spawn")
with ctx.Pool(4) as p:
p.map(worker, [1, 2, 3, 4])
logging.info("stop")
spawn or forkserver start methods result in the following output:
2020-10-14 23:05:36,979: sleepi2020-10-14 23:05:41,030: stop
fork start method results in the following output:
2020-10-14 23:09:59,094: start
2020-10-14 23:09:59,103: sleeping 1 seconds
2020-10-14 23:09:59,103: sleeping 2 seconds
2020-10-14 23:09:59,104: sleeping 3 seconds
2020-10-14 23:09:59,105: sleeping 4 seconds
2020-10-14 23:10:03,169: stop
There's a PR that may allow other startmethods but I haven't reviewed it. I'll update the documentation for a quick fix.
The approach of this module does not allow for other start methods. Please see #49 for more details.