Delgan
child process (spawn context): set parent logger globally for all modules
Thanks for this awesome library.
I would like to set the parent logger globally in a child process (spawn method) so I don't have to do that within all my modules like I saw in this example: loguru multiprocess.
The only way I found is by using the code below but I think this is not recommended since _core is not in the public api:
Module A:
from loguru import logger
...
def entrypoint_child_process(self, logger_):
logger._core = logger_._core
Module B:
from loguru import logger
...
def child_job(self):
logger.info("test")
What could be the solution in order to set the arg logger_ globally in the child process so I don't have to do the trick describe in your documentation in each module ?
UPDATE: Maybe this could be accepted ?
logger.__dict__ = logger_.__dict__.copy()
Sorry for not answering earlier. The fact is that I didn't have a solution, and this ticket made me realize that there was room for improvement in the way multiprocessing is handled.
The __dict__ update seems to be a working solution, but it's probably not very elegant.
I thought about it a little bit and I guess there is no other solution than to provide a new method to reload the logger from another one.
from multiprocessing import Process
from loguru import logger
def entrypoint_child_process(self, logger_):
logger.reload(logger_)
logger.info("Test")
if __name__ == "__main__":
logger.remove()
logger.add("file.log", enqueue=True)
process = Process(target=entry_child_process, args=(logger, ))
process.start()
process.join()
I'm also disappointed that the logger object must be passed as an argument to Process() or map() on Windows or when the "spawn" context is used, but I haven't found a viable alternative that would avoid doing this. I am open to suggestions.
I think you can't avoid logger object to be passed as an argument to multiprocessing with spawn context since you are losing the configuration by design. It's the same with std logger.
_core should be a singleton and could be updated with the logger object passed from Process/Pool...
logger.__dict__ = logger_.__dict__.copy() won't work for logger created from logger.bind in the child process.
Your suggestion to create a reload logger method for this purpose seems good.
Yes, I've thought about it in more detail since then, and I plan to add a logger.reinstall() method (no arguments).
While called in the child process, it will basically try to access the global logger instance and replace its _core by its own (i.e. by the logger sent by the parent).
def reinstall(self):
from loguru import logger # The global logger in child process with default handler
logger._replace_core(self._core)
When used with multiprocessing.Pool, that also implies we don't even need to define a custom initializer, we can just pass logger.reinstall this way:
with Pool(4, initializer=logger.reinstall) as pool:
results = pool.map(worker, [1, 10, 100])
It should be much more convenient to use.
I also have in mind a new configuration option to have the child process send logs to the parent via a TCP socket. This approach would eliminate the need to pass the logger object between processes. However, this is a different topic.
Am I understanding correctly that the kind of workaround suggested here is needed whenever you use 'spawn' context, not just on Windows?
@yhshin11 Yes, correct. This is because fork is the default context on Linux, but if you explicitly specify spawn, you end-up with the same situation as the Windows defaults.
Gotcha. It's minor but still a bit of a pain point on a fantastic library. Would love to see a fix if it's on the horizon