Error: EMFILE: too many open files

[nirendrashakya]

trafficstars

error [Error: EMFILE: too many open files, open '<File path>'] { errno: -4066, code: 'EMFILE', syscall: 'open', path: '<File path>', caller: 'git.clone' }

We have been using nodegit in our electron app as Git source control library. However, we have had a lot of issues lately with inconsistencies and incompatibilities with some basic features like clone with depth and so on. So we have been looking for alternative to manage our huge repository. One of the first issues that I have encountered with iso-git was it could not clone our repository. FYI, our repo is pretty big (13GB (includes git objects) with over 62k files).

I tried using basic clone option with depth=1 so that it would not need to download entire history and got the error above. Here is the code snippet:

const git = require('isomorphic-git');
const fs = require('fs');
git.plugins.set('fs', fs);
git
  .clone({
    dir: 'C:\\local\\repo',
    url: 'https://repo-url',
    singleBranch: true,
    depth: 1,
    noGitSuffix: true,
    ref: 'master'
  });

[billiegoose]

That's a big repo. I don't have a similar repo to test with, so I'm not sure I could develop a fix. Would be happy to accept a PR with a fix though! EMFILE: too many open files... my guess is it is trying to write too many files in parallel and needs some concurrency limit introduced.

Try wrapping fs.writeFile to introduce a concurrency limit there using a library like p-queue.

[Steveorevo]

Just encountered this error too. It seems like it's related to the frequency of files vs size. My repo has a folder with an extensive number of subfolder/files (as is the case with node_modules).

[ctjlewis]

The issue is basically this: Serverless providers restrict the number of files that can be open at once, and if you clone a repo with a directory that contains many files, you can surpass that easily. Somehow providing graceful-fs as fs does not work either, despite it being specifically designed for this use case.

This has been quite the rabbit hole, is a show-stopper for what I'm working on, and seems pretty nontrivial to work around - I want a serverless function to clone some data with isomorphic-git and do stuff with it, but there's no way to do this without getting EMFILE error.

The solution is to add a concurrency limit, so we can throttle how quickly files are opened.

[MarkiyanPyts]

Facing this as well

[jcubic]

@MarkiyanPyts You're welcome to contribute a fix.