buildcache icon indicating copy to clipboard operation
buildcache copied to clipboard

Optimize file I/O on Windows

Open mbitsnbites opened this issue 4 years ago • 0 comments

Very interesting talk about file system optimizations on Windows: "NTFS really isn't that bad" - Robert Collins (LCA 2020)

Around 20:15 Robert mentions a solution that avoids blocking on CloseHandle() by doing all the calls in a few asynchronous threads (apparently CloseHandle() blocks while virus scanners and indexing services etc do their work):

https://youtu.be/qbKGw8MQ0i8?t=1215

image

Note: Creating a single thread costs in the order of 0.05 ms on Windows (Benchmarking OS primitives), so creating a CloseHandle worker-pool of 2-3 threads at BuildCache process startup should have negligible overhead.

mbitsnbites avatar Nov 24 '21 13:11 mbitsnbites