icrawler
icrawler copied to clipboard
The interval for icrawler
First, I want to say thank you. The icrawler does have me a lot. I have read the source code, but didn't find a way to specify the interval for crawling a website. Is it possible to specify the interval for icrawler?
Currently there is no minimal interval for crawling sites. You may add some time limitation in Feeder, Parser or Downloader.