backstop-crawl icon indicating copy to clipboard operation
backstop-crawl copied to clipboard

Add option to pass configuration to simplecrawler directly

Open minorOffense opened this issue 6 years ago • 1 comments

There are a lot of options in simplecrawler. Might be useful to allow passing options directly to the crawler for really custom setups. Like limiting crawl depth or number of pages to crawl, etc.

minorOffense avatar Jun 18 '19 19:06 minorOffense

+1 I had a similar thought when writing the patch for maxDepth https://github.com/fffunction/backstop-crawl/pull/34

aaronbauman avatar Aug 31 '20 17:08 aaronbauman