Provide ability to pass ownership of buffer during read/write file operations + registration inside io_uring

[andrii0lomakin]

I am using gloomio in a disk cache (which is part of a bigger database project) which provides the possibility to load pages from the disk, keep them inside of the disk cache and change them.

Unfortunately, it is impossible to read data from a file by providing your own buffer(passing the ownership). As result, I am bound to read the page and then copy it inside a newly allocated memory chunk.

From my point of view approach provided by monio pub async fn read_at<T: IoBufMut>(&self, buf: T, pos: u64) -> crate::BufResult<usize, T> , pub async fn read_exact_at<T: IoBufMut>(&self, mut buf: T, pos: u64, ) -> crate::BufResult<(), T> and pub async fn write_at<T: IoBuf>(&self, buf: T, pos: u64) -> crate::BufResult<usize, T> https://github.com/bytedance/monoio/blob/master/monoio/src/fs/file.rs provides greater flexibility and allows to provide different types of allocators and memory management.

Would be good to have such methods with the ability to registry memory chunks inside io_uring using the method on glommio package level.

P.S. I have the impression that glommio is an absolutely great framework which suffers a bit from the fact that it was created specifically for a concrete project. Thank you for all the work which you made.

[glommer]

Hey Andrey,

When I wrote read_at and friends I did try to have an API that would take a buffer, and also tried to create traits around our buffers. So the desire to make this generic was there from the start =)

I wasn't very happy with the end result so I canned it.

Memory registration is also interesting, but in practice it has to be done when the executor comes online. After there are requests flowing it gets very hard, for reasons related to how those operations work.

I like the direction the API above is going, but doesn't it make buffer sharing hard? (many reads returning the same buffer internally).

[HippoBaro]

Memory registration is also interesting, but in practice, it has to be done when the executor comes online. After there are requests flowing it gets very hard, for reasons related to how those operations work.

We could rely on IO_USING_DRAIN to safely do so without much-added logic on our side. We should also consider the design of tokio-uring that has gone in another direction: https://github.com/tokio-rs/tokio-uring/blob/design-doc/DESIGN.md#buffer-management

[bryandmc]

I have been playing around with what the right abstraction is for this stuff, and I think we can all basically agree that we want to be able to accept one-off buffers, and register pools/custom allocators or similar. For example, I have been attempting to finish up a patch that adds back fastpoll sockets that use "buffer selection" which requires registering buffers with the kernel. Basically this is starting to sound like we either need a trait that can describe everything we need, while allowing the implementation to be radically different. So for us to use a buffer, the amount of information can vary.

They all would require:

• root ptr
• length
• capacity? (useful for when you might want to read into uninitialized memory, etc)

While others might also require:

• buffer pool ID (buffer select)
• fixed buffer ID
• ...???

One other "want" I have is that we make it somewhat pluggable. For example, allowing the user to create their own pool of buffers and pass THAT into the runtime. I think that's better, in comparison to just calling API's on the runtime and letting it manage it internally. That way it can be overridden in the future.

TLDR: I agree we need to evolve our API's to make buffer usage as flexible as possible without sacrificing performance. I really think the tokio design is closest to what I envisioned.