osx-sign icon indicating copy to clipboard operation
osx-sign copied to clipboard

fix: Serialize walkAsync so isBinaryFile doesn't throw EMFILE: too many open files.

Open amiller-gh opened this issue 1 year ago • 3 comments

Hey! Thanks for the great library – ran into an issue and have a small fix proposal.

I know this has already been raised here: https://github.com/electron/osx-sign/issues/248

However, for #reasons we need to disable ASAR packaging for our product. This means that walkAsync has to crawl all of node_modules looking for binary files. Because getFilePathIfBinary actually opens and reads the first few thousand bytes of every file, the recursive Promise.all quickly throws with EMFILE: too many open files, no matter how high I set the open file limit via ulimit.

Conveniently, just putting things into a simple for loop fixes things by serializing the file reads, giving Node a chance to release each file before reading the next one. IIRC, this shouldn't impact performance much because of how Node's FS access works. Anecdotally, we also have this change deployed to our CI and it's performing very quickly without any errors!

amiller-gh avatar Jun 26 '23 07:06 amiller-gh