wwwclient
wwwclient copied to clipboard
sebastien
Advanced web browsing, scraping and automation
The default client sometimes generates a `socket.timeout: timed out` when the connection times out. Clients should have a retry policy that retries 5 times with an increasing delay, specified in...
``` Traceback (most recent call last): File "washingtonpost.py", line 88, in testGetArticle a = self.OBJ.getArticle('http://www.washingtonpost.com/opinions/a-lifelong-love-of-la-chaumiere-in-georgetown-and-beyond/2012/03/19/gIQAiky8PS_story.html') File "washingtonpost.py", line 51, in getArticle article = Session(personality="firefox").get(url).query("#article") File "/home/edouard/ffctn/wwwclient/Sources/wwwclient/browse.py", line 663, in get...
Sometimes when I scrape a big text: ``` Traceback (most recent call last): File "/home/edouard/ffctn/FF-Collector/Sources/collector/commands/washingtonpost.py", line 91, in testGetArticle r = self.OBJ.getResults('weather', 3, True) File "/home/edouard/ffctn/FF-Collector/Sources/collector/commands/washingtonpost.py", line 43, in getResults...