Retry API requests which failed due to whatever reasons

[superbrothers]

trafficstars

GitHub API sometimes returns 502 error. github-backup fails by one that error.

github-back currently supports retry requests which failed due to connection times out and rate-limiting. In addition these, I hope that github-backup supports retry requests which failed due to whatever reasons such as 502 error.

[josegonzalez]

Pull requests welcome.

[evandrocoan]

I got this stack trace sometimes, is this covered?

...
Retrieving evandrocoan labels
Requesting https://api.github.com/repos/evandrocoan/AmxxSimpleIde/labels?per_page=100&page=1
Traceback (most recent call last):
  File "/usr/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/lib/python3.5/http/client.py", line 1252, in connect
    super().connect()
  File "/usr/lib/python3.5/http/client.py", line 849, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib/python3.5/socket.py", line 711, in create_connection
    raise err
  File "/usr/lib/python3.5/socket.py", line 702, in create_connection
    sock.connect(sa)
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/Professional/.local/bin/github-backup", line 444, in _get_response
    r = urlopen(request)
  File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/usr/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 1297, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.5/urllib/request.py", line 1256, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 22] Invalid argument>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/Professional/.local/bin/github-backup", line 1019, in <module>
    main()
  File "/home/Professional/.local/bin/github-backup", line 1014, in main
    backup_repositories(args, output_directory, repositories)
  File "/home/Professional/.local/bin/github-backup", line 681, in backup_repositories
    backup_labels(args, repo_cwd, repository, repos_template)
  File "/home/Professional/.local/bin/github-backup", line 845, in backup_labels
    label_cwd)
  File "/home/Professional/.local/bin/github-backup", line 982, in _backup_data
    data = retrieve_data(args, template)
  File "/home/Professional/.local/bin/github-backup", line 403, in retrieve_data
    r, errors = _get_response(request, auth, template)
  File "/home/Professional/.local/bin/github-backup", line 449, in _get_response
    log_warning(e.reason)
  File "/home/Professional/.local/bin/github-backup", line 68, in log_warning
    for msg in message:
TypeError: 'OSError' object is not iterable

[einsteinx2]

I'm currently looking into this logic right now due to some other changes I was making. I'll be doing a bunch of refactoring to the error handling as I found a lot of room for improvement. Expect incoming PRs relatively soon to more gracefully handle errors (possibly with command line options to choose whether to stop on error or continue)

[jacekn]

Another one that might be worth retrying is:

  File "/usr/local/lib/python3.5/dist-packages/github_backup/github_backup.py", line 798, in backup_repositories
    backup_pulls(args, repo_cwd, repository, repos_template)
  File "/usr/local/lib/python3.5/dist-packages/github_backup/github_backup.py", line 927, in backup_pulls
    pulls[number]['comment_data'] = retrieve_data(args, template)
  File "/usr/local/lib/python3.5/dist-packages/github_backup/github_backup.py", line 493, in retrieve_data
    return list(retrieve_data_gen(args, template, query_args, single_request))
  File "/usr/local/lib/python3.5/dist-packages/github_backup/github_backup.py", line 476, in retrieve_data_gen
    response = json.loads(r.read().decode('utf-8'))
  File "/usr/lib/python3.5/http/client.py", line 471, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.5/http/client.py", line 619, in _safe_read
    raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(10102 bytes read, 7990 more expected)