sentry-python icon indicating copy to clipboard operation
sentry-python copied to clipboard

IGNORED_LOGGERS is part of broken shutdown

Open untitaker opened this issue 5 years ago • 3 comments

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/threading.py", line 69, in run
    reraise(*_capture_exception())
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/threading.py", line 67, in run
    return old_run_func(self, *a, **kw)
  File "/usr/local/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/metrics.py", line 101, in worker
    logger.exception("Unable to incr internal metric")
  File "/usr/local/lib/python2.7/logging/__init__.py", line 1207, in exception
    self.error(msg, *args, **kwargs)
  File "/usr/local/lib/python2.7/logging/__init__.py", line 1200, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/local/lib/python2.7/logging/__init__.py", line 1293, in _log
    self.handle(record)
  File "/usr/local/lib/python2.7/logging/__init__.py", line 1303, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/logging.py", line 89, in sentry_patched_callhandlers
    if record.name not in _IGNORED_LOGGERS:
TypeError: argument of type 'NoneType' is not iterable

https://github.com/getsentry/onpremise/pull/615/checks?check_run_id=1037005000

untitaker avatar Aug 27 '20 14:08 untitaker

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

github-actions[bot] avatar Dec 23 '21 15:12 github-actions[bot]

@untitaker Do you know if this is still relevant? Is this something we should keep and fix?

antonpirker avatar Feb 25 '22 09:02 antonpirker

I think so. The fix is to keep a local reference to the IGNORED_LOGGERS variable such that it doesn't get cleared on process shutdown.

untitaker avatar Feb 28 '22 09:02 untitaker

Wanted to fix this, but can not reproduce.

There is a thread and in that thread a logger.exception() is done and that's it, right? Is this Python 2 only?

antonpirker avatar Nov 20 '23 15:11 antonpirker

@antonpirker can you tell me how you tried to repro this issue?

There is a thread and in that thread a logger.exception() is done and that's it, right? Is this Python 2 only?

the issue is that when the python process shuts down, python sets all global variables to None. so this is not thread shutdown, but when the process exits.

the repro case will be very race condition-y, but i think you want this setup:

  1. a thread that keeps logging errors or somehow calls callHandlers continuously (or multiple of them)
  2. a noop transport for speed
  3. a main thread that does nothing and just exits

untitaker avatar Nov 20 '23 15:11 untitaker

btw I don't actually know if it's python2 only, but i don't think so

untitaker avatar Nov 20 '23 15:11 untitaker

Thanks @untitaker

I tried with your suggested setup, but still could not reproduce. But with the other information, I created a fix https://github.com/getsentry/sentry-python/pull/2530

Would you review this please?

antonpirker avatar Nov 23 '23 13:11 antonpirker

yeah that's fine, those are tricky to repro. approved

untitaker avatar Nov 23 '23 16:11 untitaker