loguru
loguru copied to clipboard
Logger can't create file if multiple processes are using the same log
I have a webservice with fastapi with gunicorn. When logger tries to create files, the Exception is raised:
[14] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
File "/venv/lib/python3.9/site-packages/uvicorn/workers.py", line 66, in init_process
super(UvicornWorker, self).init_process()
File "/venv/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
self.load_wsgi()
File "/venv/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
self.wsgi = self.app.wsgi()
File "/venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/venv/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
return self.load_wsgiapp()
File "/venv/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
return util.import_app(self.app_uri)
File "/venv/lib/python3.9/site-packages/gunicorn/util.py", line 359, in import_app
mod = importlib.import_module(module)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/app/main.py", line 15, in <module>
logger.add(
File "/venv/lib/python3.9/site-packages/loguru/_logger.py", line 762, in add
wrapped_sink = FileSink(path, **kwargs)
File "/venv/lib/python3.9/site-packages/loguru/_file_sink.py", line 166, in __init__
self._initialize_file()
File "/venv/lib/python3.9/site-packages/loguru/_file_sink.py", line 186, in _initialize_file
self._file = open(path, **self._kwargs)
PermissionError: [Errno 13] Permission denied: '/app/web_service_debug_45531b7a-85a4-11ec-a73f-0242ac110002.log'
Since gunicorn starts worker per process, smth bad is happening with log files. I even tried to set the unique number for each start, but it didn't help:
from loguru import logger
pid = str(uuid.uuid1())
logger.add(
f"web_service_debug_{pid}.log",
filter=lambda record: record["level"].name == "DEBUG",
rotation="500 MB",
backtrace=True,
diagnose=True,
compression="zip", enqueue=True)
logger.add(
f"web_service_info_{pid}.log",
filter=lambda record: record["level"].name == "INFO",
rotation="200 MB",
backtrace=True,
diagnose=True, enqueue=True)
logger.add(
f"web_service_error_{pid}.log",
filter=lambda record: record["level"].name == "ERROR",
rotation="500 MB",
backtrace=True,
diagnose=True, enqueue=True)
logger.add(
f"web_service_{pid}.log",
filter=lambda record: record["level"].name in ["ERROR", "INFO", "DEBUG"],
rotation="700 MB",
backtrace=True,
diagnose=True,
compression="zip",
serialize=True, enqueue=True)
How can I user logger if the main.py is used by other processes automatically, and I can't manager them?