EMFILE: too many open files

[rperryng]

Hey there,

I'm trying to use node-folder-hash as a means of validating whether the $(yarn cache dir) folder gets modified after restoring this cache folder (in CI) and running yarn install, so that I can avoid re-saving the (large) cache folder if it's unchanged. However, I ran into this error EMFILE: too many open files error. I was able to reproduce this locally by running hashElement on a folder that contains 200 copies of expressJS

Error: EMFILE: too many open files, scandir 'express-146/node_modules/fast-json-stable-stringify/.github'

From reading the source code, I see that hashElement is pre-configured to use graceful-fs, and graceful-fs apparently deals with EMFILE: too many open files by queueing/retrying readdir calls when EMFILE is encountered.

Could this be an issue with the way node-folder-hash is using graceful-fs? The only caveat I see in their docs is that the asynchronous APIs must be used to get these benefits, but it seems only the fs.promises module is used so I would think it should be ok.

[marc136]

Thanks for reporting this @rperryng

I also could not see any place where graceful-fs is not used.

But I just noticed that it does not patch fs.promise https://github.com/isaacs/node-graceful-fs/issues/160, so it seems like since b2c9fc that might be an issue with my package.

But even if I were to change it back to using error-first-callbacks, or to promisify the fs-graceful code, I'm not sure if the issue with EMFILE could not happen, as I use fs.createReadStream (which is patched by graceful-fs, but does not handle the error).

I will try to get hold of a macbook, and look into fixing this next week.

You don't happen to have an error message that will print a line number? That way it would be easier to guess where the problem occurs.

[rperryng]

You don't happen to have an error message that will print a line number? That way it would be easier to guess where the problem occurs.

Ah, sorry about that I should have included the full error message I was getting (there wasn't too much more to it):

node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: EMFILE: too many open files, scandir 'express-146/node_modules/filelist/node_modules'] {
  errno: -24,
  code: 'EMFILE',
  syscall: 'scandir',
  path: 'express-146/node_modules/filelist/node_modules'
}

syscall: 'scandir', which I think is coming from when node-folder-hash calls fs.promises.readdir (it seems the scandir syscall comes from this "ReadDir" method in fs)

Your point about createReadStream not handling EMFILE still seems valid though (nice catch), hmm.

[marc136]

Oh, another note: The 3.x versions should not raise the EMFILE error.

[marc136]

This issue was fixed with v4.0.4 and it should no longer be the main reason for anyone to stay on the 3.x versions.

[rperryng]

Thanks for all your work! I appreciate it :)