js-crawler icon indicating copy to clipboard operation
js-crawler copied to clipboard

Published 20 hours ago • verified publisher icon antivanov

robots.txt

Open Muneem opened this issue 8 years ago • 1 comments

currently js-crawler does not look and follow robots.txt file. This feature should be added. There should be one variable to check if crawler reads and follow robots.txt or nor.

Muneem avatar Feb 22 '17 17:02 Muneem

Thank you, this can be a useful feature, will take a look a bit later.

amoilanen avatar Mar 08 '17 21:03 amoilanen