Limit files option

[aikar]

Hello, can you please provide an option for a max file limit that once X files have been found, stop further scanning operations?

I understand an operation may be in progress that returns file lists greater than the limit, so if it can just discard the excess.

Use case is scanning directories with large number of files but you only want to pull a few of them out at a time, for example a "drop/pickup" directory where external sources drop huge numbers of files into the directory and the node process scans and it and picks up work out of it, we may have millions of files but only want to process a few thousand at a time.

One primary reason is that memory usage is really high when the folder is large due to generating strings for all of the file names and holding them in memory during the operations, and would like to constrain that.

I tried to implement count limits by filtering like so but this doesn't stop the library from further work, it just more discards the work.

export async function walkDir(dir: string, filter: (file: string, isDirectory: boolean) => boolean | void, limit?: number): Promise<Array<string>> {
    let apiBuilder = (new fdir()).withFullPaths();
    if (limit) {
        let count = 0;
        apiBuilder = apiBuilder.filter(function (file, isDirectory) {
            return (isDirectory ? count < limit : count++ < limit) && (!filter || filter(file, isDirectory));
        });
    } else if (filter) {
        apiBuilder = apiBuilder.filter(filter);
    }
    return apiBuilder.crawl(dir).withPromise();
}

So a way to improve this memory wise would be nice.