loguru icon indicating copy to clipboard operation
loguru copied to clipboard

Logger can't create file if multiple processes are using the same log

Open lyriccoder opened this issue 2 years ago • 0 comments

I have a webservice with fastapi with gunicorn. When logger tries to create files, the Exception is raised:

 [14] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/venv/lib/python3.9/site-packages/uvicorn/workers.py", line 66, in init_process
    super(UvicornWorker, self).init_process()
  File "/venv/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/venv/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/venv/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/venv/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/venv/lib/python3.9/site-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/app/main.py", line 15, in <module>
    logger.add(
  File "/venv/lib/python3.9/site-packages/loguru/_logger.py", line 762, in add
    wrapped_sink = FileSink(path, **kwargs)
  File "/venv/lib/python3.9/site-packages/loguru/_file_sink.py", line 166, in __init__
    self._initialize_file()
  File "/venv/lib/python3.9/site-packages/loguru/_file_sink.py", line 186, in _initialize_file
    self._file = open(path, **self._kwargs)
PermissionError: [Errno 13] Permission denied: '/app/web_service_debug_45531b7a-85a4-11ec-a73f-0242ac110002.log'

Since gunicorn starts worker per process, smth bad is happening with log files. I even tried to set the unique number for each start, but it didn't help:

from loguru import logger

pid = str(uuid.uuid1())
logger.add(
    f"web_service_debug_{pid}.log",
    filter=lambda record: record["level"].name == "DEBUG",
    rotation="500 MB",
    backtrace=True,
    diagnose=True,
    compression="zip", enqueue=True)
logger.add(
    f"web_service_info_{pid}.log",
    filter=lambda record: record["level"].name == "INFO",
    rotation="200 MB",
    backtrace=True,
    diagnose=True, enqueue=True)
logger.add(
    f"web_service_error_{pid}.log",
    filter=lambda record: record["level"].name == "ERROR",
    rotation="500 MB",
    backtrace=True,
    diagnose=True, enqueue=True)
logger.add(
    f"web_service_{pid}.log",
    filter=lambda record: record["level"].name in ["ERROR", "INFO", "DEBUG"],
    rotation="700 MB",
    backtrace=True,
    diagnose=True,
    compression="zip",
    serialize=True, enqueue=True)

How can I user logger if the main.py is used by other processes automatically, and I can't manager them?

lyriccoder avatar Feb 04 '22 10:02 lyriccoder