multiprocessing-logging icon indicating copy to clipboard operation
multiprocessing-logging copied to clipboard

Add support for `spawn` and `forkserver` start methods

Open apriha opened this issue 4 years ago • 1 comments

Thanks for the library! As of Python 3.8, the default start method for multiprocessing processes on macOS is spawn; see here.

Unfortunately, as noted in #40 and #28, the current implementation of multiprocessing-logging does not support the spawn start method. It seems forkserver has the same type of issue.

Suggestion is to add support for spawn and forkserver methods, or update documentation to note that the fork start method should be used.


Example to reproduce:

import logging
import multiprocessing
import time

from multiprocessing_logging import install_mp_handler

logging.basicConfig(
    filename="output.txt",
    format="%(asctime)s: %(message)s",
    filemode="w",
    level=logging.INFO,
)
install_mp_handler()


def worker(x):
    logging.info(f"sleeping {x} seconds")
    time.sleep(x)


if __name__ == "__main__":
    logging.info("start")

    # https://docs.python.org/3/library/multiprocessing.html#multiprocessing.get_context
    ctx = multiprocessing.get_context("spawn")
    with ctx.Pool(4) as p:
        p.map(worker, [1, 2, 3, 4])

    logging.info("stop")

spawn or forkserver start methods result in the following output:

2020-10-14 23:05:36,979: sleepi2020-10-14 23:05:41,030: stop

fork start method results in the following output:

2020-10-14 23:09:59,094: start
2020-10-14 23:09:59,103: sleeping 1 seconds
2020-10-14 23:09:59,103: sleeping 2 seconds
2020-10-14 23:09:59,104: sleeping 3 seconds
2020-10-14 23:09:59,105: sleeping 4 seconds
2020-10-14 23:10:03,169: stop

apriha avatar Oct 15 '20 06:10 apriha

There's a PR that may allow other startmethods but I haven't reviewed it. I'll update the documentation for a quick fix.

jruere avatar Oct 15 '20 16:10 jruere

The approach of this module does not allow for other start methods. Please see #49 for more details.

jruere avatar Dec 20 '22 06:12 jruere