SearchingReddit icon indicating copy to clipboard operation
SearchingReddit copied to clipboard

Published 20 hours ago • verified publisher icon eleweek

Add retries to crawler

Open eleweek opened this issue 10 years ago • 2 comments

In the episode 6 (http://www.youtube.com/watch?v=dAp0GkWSxaA) I noticed that crawler crashes when it tries to get current_page (current_page = downloadRedditUrl(current_page_url)). It would be nice to have retries here.

eleweek avatar Dec 20 '14 22:12 eleweek

this gives a simple workaround for the 504 HTTP Error 504 Gateway timeout , from the link, "This problem is entirely due to slow IP communication between back-end computers, possibly including the Web server. Only the people who set up the network at the site which hosts the Web server can fix this problem."

CheoR avatar Jan 13 '15 05:01 CheoR

Hm, this issue may still be relevant in certain parts. The project uses reddit api/praw now though

eleweek avatar Mar 16 '15 21:03 eleweek