sunder icon indicating copy to clipboard operation
sunder copied to clipboard

The default behavior of `std::read_all` is slow for large files

Open ashn-dot-dev opened this issue 8 months ago • 0 comments

Discovered on commit 064e8ab19b64664729c1d39145225e9012bca267. The function std::read_all_with_allocator will buffer data 4096 bytes at a time and will call std::slice[[byte]]::resize_with_allocator for each buffered read. The default std::global_allocator (of type std::general_allocator) behaves poorly when reallocations occur every 4096 bytes, leading to poor performance when multi-MB files are read with std::read_all.

Changing the behavior of std::read_all_with_allocator would be difficult, as a valid use case for std::read_all_with_allocator could be to dynamically allocate a byte buffer exactly big enough to store the file contents, and then read the file with a std::linear_allocator.

ashn-dot-dev avatar Jun 10 '24 15:06 ashn-dot-dev