spidy icon indicating copy to clipboard operation
spidy copied to clipboard

Published 20 hours ago • verified publisher icon twiny

Crawl only deeper not upwards in hierarchy

Open BennyWindolph opened this issue 2 years ago • 0 comments

Is it possible to only crawl deeper and not upwards in hierarchy or sideways?

So when i provide the url https://test.com/test1/test2/test3/ that the crawler only crawls everything under /test3/ but not things under /test2/ or higher?

BennyWindolph avatar Mar 16 '23 13:03 BennyWindolph