crawl-anywhere icon indicating copy to clipboard operation
crawl-anywhere copied to clipboard

Published 20 hours ago • verified publisher icon bejean

Recrawl strategy

Open bejean opened this issue 11 years ago • 0 comments

Create a fast recrawl option. This option could allow to recrawl a web site often an quickly by crawling only at a maximum depth of 1 or 2 levels for declared starting urls and declared RSS feed. Memorize discovered RSS feeds during crawls in order to be used during next fast crawls.

bejean avatar May 31 '13 11:05 bejean