raven-python icon indicating copy to clipboard operation
raven-python copied to clipboard

Logging timeouts in Sentry with Gunicorn

Open dunkelstern opened this issue 6 years ago • 10 comments

Hi,

Would there be interest in logging timeout exceptions to Sentry?

I happen to just have written a custom gunicorn worker (sync mode) that does exactly that. If you use that all timeouts that happen will be logged as a Timeout-exception with stacktrace to sentry.

Should I consider to make a pull request or is that out of scope for raven-python?

dunkelstern avatar Jan 18 '18 11:01 dunkelstern

@dunkelstern i think it would be in scope. I did something like this in the past from a background thread. I think however it would have to work in a way where gunicorn is not required to make sense in the sdk.

mitsuhiko avatar Jan 18 '18 11:01 mitsuhiko

This would be a bit complicated as gunicorn kills workers that are stuck too long, so the wsgi app below the worker will just end without any notification. If you implement a signal handler in the worker you have the possibility to send out a sentry message before dieing.

dunkelstern avatar Jan 26 '18 11:01 dunkelstern

This doesn't seem ideal, but might another possible approach be to set a timeout inside the app that is a bit shorter than the gunicorn timeout, and then log the current stack after that shorter time (and then letting gunicorn kill the worker as usual later on)? One advantage of this approach is that you would be able to distinguish timeouts from signals happening for potentially other reasons.

cjerdonek avatar Feb 02 '18 11:02 cjerdonek

Is there any update on this issue since the last 1.5 years? I'd be very important to log [CRITICAL] WORKER TIMEOUT issues in Sentry in production. Also, I found no way to get the URL / path out of such lines, they are simply not showing anything. The best way I found was to check the nginx logs for error code 499.

hyperknot avatar Sep 23 '19 18:09 hyperknot

Might be worth revisiting in the new SDK. We're unlikely going to add new features to the legacy raven client.

mitsuhiko avatar Sep 23 '19 19:09 mitsuhiko

Considering that the new SDK sends any log that is error or above to Sentry, I believe this already works.

https://github.com/getsentry/sentry-python

untitaker avatar Sep 23 '19 19:09 untitaker

We are using the new SDK and it definitely didn't send anything to Sentry. Should I open an issue in the new repo?

hyperknot avatar Sep 23 '19 19:09 hyperknot

Yes please

untitaker avatar Sep 23 '19 19:09 untitaker

@hyperknot I actually know what the issue is. Gunicorn spawns multiple workers which initialize the SDK internally, but your workers are dead. You can probably fix this by creating a Python module where the SDK is initialized, and passing that as config file using gunicorn -c python:module.name.of.config

untitaker avatar Sep 23 '19 19:09 untitaker

@untitaker I've opened issue: https://github.com/getsentry/sentry-python/issues/510

hyperknot avatar Sep 23 '19 19:09 hyperknot