node-klaw

node-klaw copied to clipboard

RangeError: Maximum call stack size exceeded at node_modules/klaw/src/index.js:45:23 at go$readdir$cb (node_modules/graceful-fs/graceful-fs.js:187:14) at FSReqWrap.oncomplete (fs.js:135:15)

version 2.1.1 but nothing in new version relates here.

I have a folder with 200k+ files in it that crashes klaw.

        const crawl = async (subdir: string, cb) => {
            const done = resolvable();
            klaw(path.join(dir, subdir), {
                filter: (item) => {
                    return item.endsWith(".job");
                }
            }).on('data', (item) => {
                cb(item.path);               
            }).on('end', () => done.resolve());
            await done;
        };

Ended up switching to https://www.npmjs.com/package/walk due to this bug.

Hey @jprichardson how would you feel changing this https://github.com/jprichardson/node-klaw/blob/master/src/index.js#L53 to not use apply? Either spreading the array or concat will avoid stack size range error and also probably would be faster?

i think concat might probably be the first solution but then array will need to be mutated

Certainly. Should modernize the code as well (separate PR)

Thanks for the quick response. PR opened :)

@oorestisime would you be willing to put a test in place verifying that this actually fixes the problem? Thanks.

I can take a look yeah. but this means adding a test with more than 200k files or so (haven't looked how the tests are )

Couldn't a test harness be used to generate the files?

We experienced this problem with around 60K files in a folder.

hey i am back at this. i completely forgot about it. what are our options here. change seems fairly trivial imho

@oorestisime's fix it working for us with 260k pages.