robots.js
robots.js copied to clipboard
HTTP request User-Agent header styled incorrectly
When providing a string user agent to the constructor or allowing the default user agent string, it is built as
"headers":{"userAgent":"<user agent string>"}
Servers do not log this header; they show the user agent field as "-". The option needs to be styled as
"headers":{"User-Agent":"<user agent string>"}
Workaround: Pass an object to constructor instead:
var parser = new robots.RobotsParser(
'http://nodeguide.ru/robots.txt',
{ "headers": { "User-Agent": "My User Agent String" } },
after_parse
);