robotstxt
robotstxt copied to clipboard
sitemap causes an exception
I get an exception when parsing the following robots.txt. robots.txt:
Sitemap: http://rebeccasugar.tumblr.com/sitemap1.xml
User-agent: *
Disallow: /private
Disallow: /random
Disallow: /day
Crawl-delay: 1
site: http://rebeccasugar.tumblr.com/
/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:305
return myGateKeeper.sitemaps.push({
^
TypeError: Cannot read property 'sitemaps' of undefined
at RobotsTxt.<anonymous> (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:305:34)
at __bind (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:3:63)
at RobotsTxt.parse (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:377:9)
at RobotsTxt.__bind [as parse] (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:3:63)
at RobotsTxt.crawl.handler.request.txt (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:235:25)
at IncomingMessage.__bind (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:3:63)
at IncomingMessage.EventEmitter.emit (events.js:126:20)
at IncomingMessage._emitEnd (http.js:366:10)
at HTTPParser.parserOnMessageComplete [as onMessageComplete] (http.js:149:23)
at Socket.socketOnData [as ondata] (http.js:1367:20)