Automatically resume and slow down on timeouts

[emijrp]

From [email protected] on November 05, 2013 20:35:07

wiki-site.com has some nasty configuration for which it stops responding, or sends error 403, if you do too many requests. I've tried 60, now 90 seconds delay and will probably find one which works with them in end, but it would be nice to have a better handling of this error than the current one:

Traceback (most recent call last): File "dumpgenerator.py", line 1195, in main() File "dumpgenerator.py", line 1152, in main config, other = getParameters(params=params) File "dumpgenerator.py", line 908, in getParameters if checkAPI(config['api']): File "dumpgenerator.py", line 931, in checkAPI f = urllib2.urlopen(req) File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib/python2.6/urllib2.py", line 1170, in http_open return self.do_open(httplib.HTTPConnection, req) File "/usr/lib/python2.6/urllib2.py", line 1145, in do_open raise URLError(err) urllib2.URLError: <urlopen error [Errno 110] Connection timed out> No tag found: dump failed, needs fixing; resume didn't work. Exiting.

Original issue: http://code.google.com/p/wikiteam/issues/detail?id=70

[emijrp]

From [email protected] on November 05, 2013 11:42:53

Labels: -Type-Defect -Priority-Medium Type-Enhancement Priority-Low

[emijrp]

From [email protected] on November 05, 2013 11:43:10

Blocking: wikiteam:33

[emijrp]

From [email protected] on November 08, 2013 14:40:39

Well, and maybe also urllib2.HTTPError: HTTP Error 503: Service Temporarily Unavailable which I get sometimes from them too.

[nemobis]

This should be largely fixed with https://github.com/WikiTeam/wikiteam/commit/142b48cc6926a0ed4b3e0a97a1040932d0ec7c87 but we'll see how well requests does.