robots.js icon indicating copy to clipboard operation
robots.js copied to clipboard

HTTP request User-Agent header styled incorrectly

Open midnightfreddie opened this issue 7 years ago • 0 comments

When providing a string user agent to the constructor or allowing the default user agent string, it is built as

"headers":{"userAgent":"<user agent string>"}

Servers do not log this header; they show the user agent field as "-". The option needs to be styled as

"headers":{"User-Agent":"<user agent string>"}

Workaround: Pass an object to constructor instead:

var parser = new robots.RobotsParser(
	'http://nodeguide.ru/robots.txt',
	{ "headers": { "User-Agent": "My User Agent String" } },
	after_parse
);

midnightfreddie avatar Nov 09 '17 10:11 midnightfreddie