fs-jetpack icon indicating copy to clipboard operation
fs-jetpack copied to clipboard

Use fast-glob

Open jasonkuhrt opened this issue 3 years ago • 3 comments

On Nexus we discovered that fs-jetpack's find was very slow. When we investigated why, it turned out to be because ignores were only being executed as an in-memory filter, not actually changing how the disk was traversed (IO). This had huge consequences (toward a full 1000ms of latency on project boot).

We solved our problems by switching to fast-glob.

The key for us was https://github.com/mrmlnc/fast-glob#how-to-exclude-directory-from-reading.

Given filters are actually affecting the IO.

Maybe there are some downsides I haven't thought of but right now I'm pretty sure that it would be great for fs-jetpack to change its find engine to it.

jasonkuhrt avatar Nov 30 '20 13:11 jasonkuhrt

Here is the commit where we switched away from fs-jetpack find https://github.com/prisma-labs/graphql-framework-experiment/commit/47b09117b36bba8a95a141d8a692574bcf7e5613

jasonkuhrt avatar Nov 30 '20 13:11 jasonkuhrt

The use case is that you have a big directory nested somewhere inside the directory that the find was performed on, and you don't want to touch that big directory? Or is this something more nuanced?

szwacz avatar Jan 19 '21 14:01 szwacz

@szwacz yes I think that's it

jasonkuhrt avatar Jan 19 '21 14:01 jasonkuhrt