robotstxt icon indicating copy to clipboard operation
robotstxt copied to clipboard

sitemap causes an exception

Open evilpacket opened this issue 12 years ago • 0 comments

I get an exception when parsing the following robots.txt. robots.txt:

Sitemap: http://rebeccasugar.tumblr.com/sitemap1.xml

User-agent: *
Disallow: /private
Disallow: /random
Disallow: /day
Crawl-delay: 1
site: http://rebeccasugar.tumblr.com/

/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:305
              return myGateKeeper.sitemaps.push({
                                 ^
TypeError: Cannot read property 'sitemaps' of undefined
    at RobotsTxt.<anonymous> (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:305:34)
    at __bind (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:3:63)
    at RobotsTxt.parse (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:377:9)
    at RobotsTxt.__bind [as parse] (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:3:63)
    at RobotsTxt.crawl.handler.request.txt (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:235:25)
    at IncomingMessage.__bind (/Users/adam_baldwin/Documents/projects/millimeter/node_modules/robotstxt/index.js:3:63)
    at IncomingMessage.EventEmitter.emit (events.js:126:20)
    at IncomingMessage._emitEnd (http.js:366:10)
    at HTTPParser.parserOnMessageComplete [as onMessageComplete] (http.js:149:23)
    at Socket.socketOnData [as ondata] (http.js:1367:20)

evilpacket avatar Nov 19 '12 03:11 evilpacket