icrawler icon indicating copy to clipboard operation
icrawler copied to clipboard

Published 20 hours ago • verified publisher icon hellock

The interval for icrawler

Open h0nestliar opened this issue 6 years ago • 1 comments

First, I want to say thank you. The icrawler does have me a lot. I have read the source code, but didn't find a way to specify the interval for crawling a website. Is it possible to specify the interval for icrawler?

h0nestliar avatar Feb 19 '19 01:02 h0nestliar

Currently there is no minimal interval for crawling sites. You may add some time limitation in Feeder, Parser or Downloader.

hellock avatar Feb 21 '19 00:02 hellock