huntsman icon indicating copy to clipboard operation
huntsman copied to clipboard
verified publisher icon missinglink

Support for robots.txt

Open missinglink opened this issue 12 years ago • 0 comments

Obey robots.txt. Minimum functonality:

cancel all requests which globally disallow the huntsman User-Agent

User-agent: huntsman
Disallow: /

cancel all requests for urls which match Disallow statements

User-agent: huntsman
Disallow: /private/area

missinglink avatar Sep 22 '13 15:09 missinglink