ziglang
std.fs.File: improve the vector based operations to operate on OS-agnostic data types
Right now for example fs.File.writev looks like this:
pub fn writev(self: File, iovecs: []const os.iovec_const) WriteError!usize {
The reasoning here is that we would need to heap allocate if we needed to convert from []const []const u8 to the underlying data type the OS wants.
However this breaks our abstractions because now the callsites must reach into std.os and I think we don't have this hooked up to work on Windows for example.
I think we can get everything we want - we need something like this:
const Vector = extern struct {
internal_data: if (windows) something else std.os.iovec,
pub fn set(v: Vector, slice: []u8) void {
// os-specific logic to set internal data
}
};
As long as internal data matches the target OS vector for writev,readv, etc, this should work fine. Point being that the callsite can stack-allocate arrays of this Vector and it will pointer-cast properly inside fs.File.{write,read}v functions. Then we have abstracted over different layouts for different targets.
It should probably be called something other than Vector since that already has to do with SIMD, and also sometimes that is what array lists are named.
One idea I've mentioned before is to have extern []u8 (i.e. an "extern slice") match iovec for a given target.
I've been implementing ReadFileScatter for a personal project, and the stdlib comment (as well as a few other issue mentions of ReadFileScatter/WriteFileGather) led me here. All things considered, I'm not really sure there's a reasonable stack-allocated solution for the windows side of this, even at callsite, while retaining full functionality.
Unlike iovecs and readv/writev implementations, the windows equivalents require pointer arrays based on system memory page sizes, which is usually just going to mean preemptively stack-allocating tens or hundreds of thousands of pointers (I think I was around 340k-ish pointers on a 14gig file with 4096 byte page size, though that's likely far from the norm) and slicing whichever portion of the array is actually filled with those pointers on file operation. It also requires opening the file handle with certain flags that aren't part of standard file usage (no buffering + overlapped mode), is limited to 2^32 - pagesize bytes read per call (which to be fair is a limitation of standard reads as well) requiring iterating slices and file offsets if reading more than 4gb of data and then waiting on the asynchronous operation(s) to complete.
I definitely think that ReadFileScatter and WriteFileGather could be implemented as part of stdlib but abstracting that across both operating systems would require explicitly removing the control of being able to write to multiple non-contiguous buffers of a user-defined size that readv offers on other systems (you could write to non-contiguous buffers or user-defined sizes, not both).
The most ergonomic solution I can see for implementing Scatter/Gather is ditching the non-contiguous reads altogether, via something along the lines of a type std.iovec.init(alloc, buffer_length, buffer_count) which based on OS (+ sys info calls if needed) could calculate in one go both the length of file buffer to allocate with appropriate page size padding + the length of the pointer arrays needed to represent those buffers appropriately (as well as storing the virtual cursor location in memory, potentially caching pagesize, and allocating an extra pagesize worth of memory to store overflow, which also comes with the downside of non-vector operations on the file would need to check this memcache as well). This could be used with FixedBufferAllocator if the caller wanted to remain stack-based, but I don't really see a way forward for comptime-known sizes or any reasonable ergonomics if expanding this to the windows side of things.
Genuinely, and unfortunately, the best solution might be to just manually iterate/populate the vectors in stdlib functions with basic read/write file operations (which is probably still a step better than the current windows approach of only operating on the first iovec member).
ReadFileScatter/WriteFileGather aren't usable unless each iovec's length is a page size AND the file's opened with FILE_FLAG_NO_BUFFERING AND the number of bytes requested is a multiple of the sector size of the file system. This makes them extremely niche and geared more towards batching async io than syscall efficiency.
Currently in the stdlib readv/writev are used on smallish mixed stack/heap allocated buffers for parsing and avoiding syscalls. This is incompatible.
Asking for buffers ahead of time and providing a reader/writer interface could work, but at that point why not just use a BufferedReader/BufferedWriter yourself where YOU get to choose the size?
After reading Windows docs and Rust stdlib code, I don't think there's a way to make OS-agnostic gather/scatter vector types. Rust just calls read on the first iovec member unless there's an optional writev function implemented on the Reader/Writer trait. Zig could do something similar for std.fs.File and std.net.Stream, but it's sloppy second-class support.
I think this issue can be closed with the sad conclusion that readv and writev won't be able to enjoy Windows support and should not be built upon.
If someone else comes up with a solution, #19376 should be reopened and these functions should always be preferred to the plain read and write.