Error while backup large collection

[bushev]

Hello!

It seems this issue still not resolved: https://github.com/hex7c0/mongodb-backup/issues/16

Node: v7.2.1 mongodb-backup: v1.6.8

{ Error: Cannot find module 'tar'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at /Users/bushev/IdeaProjects/proj/node_modules/mongodb-backup/index.min.js:195:42
    at /Users/bushev/IdeaProjects/proj/node_modules/mongodb-backup/index.min.js:22:20
    at /Users/bushev/IdeaProjects/proj/node_modules/mongodb-backup/node_modules/graceful-fs/polyfills.js:287:18
    at FSReqWrap.oncomplete (fs.js:111:15) code: 'MODULE_NOT_FOUND' }
fs.js:557
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^

Error: EMFILE: too many open files, open '/Users/bushev/IdeaProjects/proj/logs/error.log'
    at Object.fs.openSync (fs.js:557:18)
    at Object.fs.writeFileSync (fs.js:1222:33)
    at Object.fs.appendFileSync (fs.js:1268:6)
    at process.on.err (/Users/bushev/IdeaProjects/proj/queue-server.js:50:19)
    at emitOne (events.js:96:13)
    at process.emit (events.js:188:7)
    at process._fatalException (bootstrap_node.js:292:26)

Also, Q: Why we need to create a separate file for each collection item?

[hex7c0]

hi @bushev, sorry for delay!

err is about Cannot find module 'tar', where is the relationship between large collection?

[bushev]

Hi @hex7c0!

The root cause of "Cannot find module 'tar'" is a require call fail. It fails due to EMFILE, which originated by writing a lot of separated files (instances on large collection).

[hex7c0]

sure, sorry for this issue

I'm working on it

[crh3675]

We had the same EMFILE issue but even more so, this module completely crashed our server. We have 6 CPUs and they pegged to 299% and then completely obliterated all other PM2 processes and corrupted the PM2 processes file.

We had to empty out the PM2 processes file and reboot the server - pretty awful experience. We definitely are going to look for an alternative unless this can be looked at.

[Zwimber]

+1