loguru icon indicating copy to clipboard operation
loguru copied to clipboard

Loguru error when using dask with a multiprocessing scheduler

Open david-hoffman opened this issue 3 years ago • 0 comments

I'm getting a RuntimeError when trying to log from a dask.delayed function using a logger handler with enqueue enabled.

Here's a minimal reproducible example

import sys, dask
from loguru import logger

logger.add(sys.stderr, enqueue=True)

@dask.delayed
def log():
    logger.info("Logging!")

dask.compute(*[log() for i in range(10)], scheduler="processes")

Which produces the following error:

RuntimeError: SimpleQueue objects should only be shared between processes through inheritance

I can get everything to run by importing the logger within the delayed function:

In [1]: import sys, dask
   ...: from loguru import logger
   ...: 
   ...: logger.add(sys.stderr, enqueue=True)
   ...: 
   ...: @dask.delayed
   ...: def log():
   ...:     from loguru import logger
   ...:     logger.info("Logging!")
   ...: 
   ...: dask.compute(*[log() for i in range(10)], scheduler="processes")

2022-09-13 21:42:36.001 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.001 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.001 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.001 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.002 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.002 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.004 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.004 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.004 | INFO     | __main__:log:9 - Logging!
2022-09-13 21:42:36.004 | INFO     | __main__:log:9 - Logging!

Out[1]: (None, None, None, None, None, None, None, None, None, None)

However, this means all the configured handlers are lost.

david-hoffman avatar Sep 14 '22 04:09 david-hoffman