Unhandled EMFILE

[aredridel]

:; codepaint infer "**/*.js"

path.js:309
      var path = (i >= 0) ? arguments[i] : process.cwd();
                                                   ^
Error: EMFILE, too many open files
    at Object.exports.resolve (path.js:309:52)
    at Object.module.exports.parse (/usr/local/lib/node_modules/codepainter/node_modules/editorconfig/editorconfig.js:135:19)
    at MultiInferrer.module.exports.util.inherits.onGlobPath (/usr/local/lib/node_modules/codepainter/lib/MultiInferrer.js:48:41)
    at Array.forEach (native)
    at MultiInferrer.module.exports.util.inherits.onGlobPaths (/usr/local/lib/node_modules/codepainter/lib/MultiInferrer.js:42:9)
    at /usr/local/lib/node_modules/codepainter/lib/util/index.js:24:4
    at /usr/local/lib/node_modules/codepainter/node_modules/async/lib/async.js:232:13
    at /usr/local/lib/node_modules/codepainter/node_modules/async/lib/async.js:119:25
    at /usr/local/lib/node_modules/codepainter/node_modules/async/lib/async.js:24:16
    at /usr/local/lib/node_modules/codepainter/node_modules/async/lib/async.js:229:17

[jednano]

How many files do you have in that directory? I fear if there's a node_modules folder in there or some other folder that has a huge number of files that you might run into this; though, I've not seen this error myself.

What is your environment?

[jednano]

In the meantime, you might benefit from being more specific about what you're scanning. Try something like app/**/*.js

[aredridel]

Indeed, I can work around it, probably (but I was just giving the tool a spin and don't have a driving need to)

This is on MacOS 10.9.

There are several node_modules directories; a great amount of PHP in many deep directories; a whole pile of stuff in my development environment.

[jednano]

It looks like an issue with the async dependency. I wouldn't be surprised if updating my dependencies would fix the issue here. I think I can do that tonight.

[jednano]

I just updated the dependencies. Try-out the new patch version 0.3.29.

BTW, your scenario seems like a worst-case scenario, but I'm hoping the dependency updates resolve it, as it doesn't appear to be an issue, directly, with codepainter.

[aredridel]

Nope. I doubt it's with async: this usually happens with unbounded concurrency while recursively looking through the file system. npm had the same problem for a while, and it took making a queue of requests that failed and retrying them, or limiting concurrency on the recursion.

Try not invoking the next layer of search until the previous has completed in codepainter/lib/util/index.js

[jednano]

I'm still not sure why that kind of logic should at all be the concern of codepainter. It seems like async should be the one responsible for throttling concurrent fie I/O. And how do we even know where to put the limit? I also wonder if the problem presents itself on a PC as well, because for I, not having a Mac, it would be difficult to test such things.

Your proposed solution might work some of the time, but it still seems like it won't work all the time. In your case, there is only one layer of search, as your glob is really greedy.

So far, I'm convinced you should use a less-greedy glob and be more explicit about what you want to search. Why on Earth would you want to infer your source code and mix those inferred settings with code in node_modules folders? That kind of defeats the purpose.

[aredridel]

It's just an arbitrary limit with a terrible error message, and completely handleable.

Async isn't what's doing that though -- looking a little deeper, it looks like the unbounded concurrency is in glob -- and that's the same module that bit npm: http://qiita.com/laiso/items/c16b178fd61e4ad8ec3e among other links show the error.

[aredridel]

(The same error can occur in any system with a large, deep tree. Whether it's third-party or not)

[aredridel]

The fix npm uses is to load the graceful-fs module, which adds retries on EMFILE.

[aredridel]

(See https://github.com/npm/npm/issues/3259)

[jednano]

Wouldn't it be better, then, if glob handled this logic? Would you consider opening an issue on the glob issue tracker?

[aredridel]

Possibly, but I don't think isaacs wants to have glob go monkey-patching core. It's a tough call, since the fix is ugly and boundary-escaping. (He's also the author of glob, of graceful-fs and of npm, so he'll have considered this already, since he can tweak any of the parts freely.)

The decision to push that kind of hackery out to the application level makes sense, since applications don't need to be so concerned about isolating the side-effects to their execution environment, since they 'own' it.

[jednano]

You make solid points. Still, it's going to be a pain to implement this fix, mostly due to the difficulty of testing it. I'll probably get around to fixing it when I have some time, but don't expect anything in the near future. I'll keep the ticket open until then. For now, try something less greedy?

[aredridel]

Absolutely! I'm happy to test, too!