crawl-anywhere icon indicating copy to clipboard operation
crawl-anywhere copied to clipboard

Published 20 hours ago • verified publisher icon bejean

Add ability to bypass robots.txt on a per-host basis

Open grimsa opened this issue 9 years ago • 0 comments

Currently "bypass_robots_file" property can only be configured globally. It would be useful to have the ability to configure it on a per-host basis, ideally via admin UI.

grimsa avatar Aug 17 '15 18:08 grimsa