pywebcopy icon indicating copy to clipboard operation
pywebcopy copied to clipboard

Published 20 hours ago • verified publisher icon rajatomar788

How to limit the crawling depth?

Open cfytrok opened this issue 3 years ago • 2 comments

It looks like the scan_level parameter was responsible for this earlier. But it has been deprecated. What can I do to limit depth of crawl?

cfytrok avatar Dec 19 '20 15:12 cfytrok

The crawl length feature was removed as it was incompatible with new algorithm. You can try to subclass the crawler class and see what happens.

rajatomar788 avatar Dec 23 '20 16:12 rajatomar788

What does that mean @rajatomar788 since getting the whole website like wget is the main feature

BradKML avatar Apr 02 '23 16:04 BradKML