light-speed-io
light-speed-io copied to clipboard
Register buffers
In summary: io_uring allows users to register buffers. The max size per buffer is 1 GiB. Read operations can read into part of a large buffer. So I think that, for example, if you had 10 x 100 MiB files to read, then you could first register a single 1 GiB buffer, and read your 10 files into different parts of that buffer.
Further reading
- https://man7.org/linux/man-pages/man2/io_uring_register.2.html - this page has lots of info about registering buffers.
- https://man7.org/linux/man-pages/man3/io_uring_prep_read_fixed.3.html
- https://man7.org/linux/man-pages/man3/io_uring_register_buffers.3.html
Thoughts about when to register buffers
I could be wrong but I think that registering buffers will really come into its own when we re-use buffers (issue #38) (there is overhead to registering buffers. That overhead will be worth it when we re-use buffers, or if we use a single registered buffer for multiple I/O operations).
Reading multiple files into a single, large buffer
We could use a single large buffer to read multiple files into. This would be most convenient when users provide a batch of operations in one go (#32), so LSIO can scan through all the requested IO operations, and figure out what buffers it requires ahead-of-time.
When using the ObjectStore
API, maybe users could initialise the io_uring worker with some hint as to how many I/O operations they plan to request, and the max size of each operation (so LSIO can register appropriate buffers).