Cache-less fast file search by directly reading the file system

[valaphee]

There is a way on Windows, and definitely also on Linux, to directly read the file system structure, which don't involve the operating systems file caching, search indices, etc.

And when searching for a file over the entire disk, it's way faster to just read for example the NTFS Master File Table; As a proof on concept, I created a project which currently implements this kind of file search for Windows NTFS file system: https://github.com/valaphee/curtain/blob/main/storage/examples/search.rs, and it is only limited by the speed of how fast the Master File Table can be read, and only takes about 1 second on my PC when searching on an 512GB NVMe volume with around 200k files.

This might also be combined with file system watches and caching. But without writing the cache to disk, as the load time of the cache is slower than just reading the file entries. And it also increases the disk wear level by an unnessary amount, especially as compressed data may completely change by just adding/removing one file.

The internals are little bit more complicated as it basically simulates the NTFS file system read operation and it also requires administrator privileges. One other downside is that it might be slower when limiting the search like for example searching for example by directory which has <= Total NTFS files / 2, which can be optimized by parsing the file structure, etc. with the cost of higher memory usage.

But its the technically the fastest way for searching files.