Errors connecting to the Rollbar API should be caught and retried

Open diranged opened this issue 9 years ago • 0 comments
Ran into this with an integration test run:
======================================================================
ERROR: integration_test_2c_execute_real (kingpin.actors.test.integration_rollbar.IntegrationRollbarDeploy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/testing.py",> line 132, in __call__
    result = self.orig_method(*args, **kwargs)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/testing.py",> line 525, in post_coroutine
    timeout=timeout)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/ioloop.py",> line 453, in run_sync
    return future_cell[0].result()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/concurrent.py",> line 232, in result
    raise_exc_info(self._exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/kingpin/actors/test/integration_rollbar.py",> line 80, in integration_test_2c_execute_real
    res = yield actor.execute()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1008, in run
    value = future.result()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/concurrent.py",> line 232, in result
    raise_exc_info(self._exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/kingpin/actors/base.py",> line 275, in _wrap_in_timer
    ret = yield gen.coroutine(f)(self, *args, **kwargs)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1008, in run
    value = future.result()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/concurrent.py",> line 232, in result
    raise_exc_info(self._exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/kingpin/actors/base.py",> line 491, in execute
    raise exceptions.ActorException(e)
ActorException: [Errno 104] Connection reset by peer
-------------------- >> begin captured logging << --------------------
kingpin.actors.rollbar.Deploy: DEBUG: [Unit Test Action] Checking for required options: ['environment', 'revision']
kingpin.actors.rollbar.Deploy: DEBUG: [Unit Test Action] Initialized (warn_on_failure=False, strict_init_context=True)
kingpin.actors.rollbar.Deploy: DEBUG: [Unit Test Action] Beginning
kingpin.actors.rollbar.Deploy: DEBUG: [Unit Test Action] Condition True evaluates to True
kingpin.actors.rollbar.Deploy: DEBUG: [Unit Test Action] kingpin.actors.rollbar.Deploy._execute() deadline: 3600(s)
kingpin.actors.rollbar.Deploy: INFO: [Unit Test Action] Sending Rollbar Deploy Notification kingpin-integration-testing/0.3.1a
kingpin.actors.rollbar.Deploy: DEBUG: [Unit Test Action] Making HTTP request to https://api.rollbar.com/api/1/deploy/ with data: comment=Integration+Tests+Are+Good%2C+MmmKay&access_token=****&local_username=Kingpin+Integration+Testing&rollbar_username=None&environment=kingpin-integration-testing&revision=0.3.1a
kingpin.actors.base: CRITICAL: Unexpected exception caught! Please contact the author (Matt Wise <[email protected]>) and provide them with this stacktrace
kingpin.actors.rollbar.Deploy: ERROR: [Unit Test Action] [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/kingpin/actors/base.py",> line 466, in execute
    result = yield self.timeout(self._execute)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1008, in run
    value = future.result()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/concurrent.py",> line 232, in result
    raise_exc_info(self._exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/kingpin/actors/base.py",> line 326, in timeout
    deadline, fut, quiet_exceptions=(exceptions.ActorTimedOut))
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1008, in run
    value = future.result()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/concurrent.py",> line 232, in result
    raise_exc_info(self._exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/kingpin/actors/rollbar.py",> line 238, in _execute
    yield self._deploy()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1008, in run
    value = future.result()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/concurrent.py",> line 232, in result
    raise_exc_info(self._exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/kingpin/actors/rollbar.py",> line 218, in _deploy
    res = yield self._fetch_wrapper(API_DEPLOY_PATH, post=escaped_post)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1008, in run
    value = future.result()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/concurrent.py",> line 232, in result
    raise_exc_info(self._exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/kingpin/utils.py",> line 220, in wrapper
    ret = yield gen.coroutine(f)(*args, **kwargs)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1008, in run
    value = future.result()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/concurrent.py",> line 232, in result
    raise_exc_info(self._exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/kingpin/actors/rollbar.py",> line 90, in _fetch_wrapper
    res = yield self._fetch(*args, **kwargs)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1008, in run
    value = future.result()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/concurrent.py",> line 232, in result
    raise_exc_info(self._exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/kingpin/actors/base.py",> line 597, in _fetch
    http_response = yield http_client.fetch(http_request)
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/gen.py",> line 1008, in run
    value = future.result()
  File "<https://XXX/job/ops-kingpin-continuous-build/ws/venv/local/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg/tornado/concurrent.py",> line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
error: [Errno 104] Connection reset by peer
--------------------- >> end captured logging << ---------------------
This should be easy to catch and retry.
Apr 26 '16 18:04 diranged