pyrollbar
pyrollbar copied to clipboard
Update Twisted integration
This PR will:
- Address https://github.com/rollbar/pyrollbar/issues/172
- Update to twisted.logger
- Handle Exceptions/Failures that come up sending a payload with Twisted
- Current system is a little fragile. It's possible to get an error stuck in the reactor forever.
Testing:
- If you're a Rollbar employee, the easiest way to cause an error to get trapped in the reactor is to try and send an error to your dev vm using the HTTPS endpoint. Twisted doesn't like self-signed certs.
Status:
- ~Attempting to reproduce some edge cases~
- ~Use
inlineCallbacksto reduce lines of code~ - Currently deciding the best way to handle failures from within Twisted itself. The easiest way to just check for that the
err.typeinlog_handleris a specific Twisted Exception or some subclass of one and don't send it back toreport_exc_info. The nicer alternative would be to somehow tag an error as coming from Rollbar but I'm not quite there yet.
@markrwilliams here's what ends up looping:
twisted.internet.task.TaskStopped:
ERROR:rollbar:2017-06-18T22:20:54.251772 Failed to post to rollbar
Traceback (most recent call last):
File "/Users/ezarowny/Development/pyrollbar/rollbar/__init__.py", line 1205, in _post_api_twisted
timeout=SETTINGS.get('timeout', DEFAULT_TIMEOUT))
RequestTransmissionFailed: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]>]
catching exception
Unexpected exception from twisted.web.client.FileBodyProducer.stopProducing
Traceback (most recent call last):
File "/Users/ezarowny/.virtualenvs/pyrollbar/lib/python2.7/site-packages/twisted/protocols/policies.py", line 125, in connectionLost
self.wrappedProtocol.connectionLost(reason)
File "/Users/ezarowny/.virtualenvs/pyrollbar/lib/python2.7/site-packages/twisted/web/_newclient.py", line 964, in dispatcher
return func(*args, **kwargs)
File "/Users/ezarowny/.virtualenvs/pyrollbar/lib/python2.7/site-packages/twisted/web/_newclient.py", line 1634, in _connectionLost_TRANSMITTING
self._currentRequest.stopWriting()
File "/Users/ezarowny/.virtualenvs/pyrollbar/lib/python2.7/site-packages/twisted/web/_newclient.py", line 865, in stopWriting
_callAppFunction(self.bodyProducer.stopProducing)
--- <exception caught here> ---
File "/Users/ezarowny/.virtualenvs/pyrollbar/lib/python2.7/site-packages/twisted/web/_newclient.py", line 194, in _callAppFunction
function()
File "/Users/ezarowny/.virtualenvs/pyrollbar/lib/python2.7/site-packages/twisted/web/client.py", line 1079, in stopProducing
self._task.stop()
File "/Users/ezarowny/.virtualenvs/pyrollbar/lib/python2.7/site-packages/twisted/internet/task.py", line 497, in stop
self._checkFinish()
File "/Users/ezarowny/.virtualenvs/pyrollbar/lib/python2.7/site-packages/twisted/internet/task.py", line 507, in _checkFinish
raise self._completionState
#172 needs to get addressed and this appears to go towards that, but I'm going to bump this to the next release because getting into the weeds of twisted is not what I want to have hold up a bunch of other changes
Fixes #163
This is a tough one dude! Very curious to see how you end up preventing that infinite loop.
With the current code, as far as I can tell, there are no infinite loops in the case of an SSL error. If you get rid of the add/remove observer stuff then you do get some looping so it appears that is part of the solution.