Lazy path-based metadata is extremely slow on Windows

[pervognsen]

Currently jwalk implements its version of DirEntry::metadata() using on-demand path-based calls to std::fs::metadata(). For programs that need stat-like metadata (like the example program which needs the file size) this leads to a per-file syscall (NtQueryAllInformationFile, roughly the equivalent of stat on Windows).

Windows provides stat-like metadata "for free" when traversing files in a directory with FindFirstFile/FindNextFile (NtQueryDirectoryFile under the hood). Since NtQueryDirectoryFile is batched via a user-space buffer, this means that you can usually enumerate and stat all files in a directory with one syscall (plus the open/close directory syscalls) or maybe a couple for large directories. Rust's own std::fs::DirEntry::metadata() is designed to take advantage of this Windows vs POSIX difference, as it's essential for baseline performance on Windows.

The result is that in one test I did (calculate the combined file size for a drive with 10^6 files and 10^5 directories) jwalk's example program runs 50% slower than a straightforward single-threaded version of the same program that avoids this pitfall. (For reference, it is 12x slower on my 8-core machine compared to a multi-threaded program that avoids this pitfall.)