newsdiffs icon indicating copy to clipboard operation
newsdiffs copied to clipboard

Published 20 hours ago • verified publisher icon ecprice

rate-limiting the scraper

Open faried opened this issue 11 years ago • 0 comments

I received an email from a local newspaper last night. They noticed the scraper (it only took them five months!), and they've asked me to rate-limit the process. I've put a time.sleep(0.3) at the end of baseparser.py's grab_url() , which should keep them happy.

I'm opening this issue so people know about the problem.

faried avatar Dec 02 '13 06:12 faried