neon icon indicating copy to clipboard operation
neon copied to clipboard

feat(pageserver): newtype for dio-aligned buffer allocation

Open yliang412 opened this issue 6 months ago • 1 comments

Problem

We need to use aligned buffer to enable Direct IO. Currently there is no primitive for constructing such buffer.

Summary of changes

  • Introduce IoBufferMut, new-type around ManuallyDrop<Vec<u8>> with custom allocation and deallocation.
  • implement bytes::BufMut trait for IoBufferMut
  • implement tokio_epoll_uring::IoBuf, tokio_epoll_uring::IoBufMut trait for IoBufferMut
  • Marker trait IoBufAlignedMut to indicate alignment.

Run miri against the unit tests, passed.

The only error emitted was unsupported operation: can't call foreign function _rjem_malloc on OS linux, this is due to miri's own limitations to support jemalloc without the unprefixed_malloc_on_supported_platforms feature on. Otherwise,

❯ cargo +nightly miri test --target x86_64-unknown-linux-gnu
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.03s
     Running unittests src/main.rs (target/miri/x86_64-unknown-linux-gnu/debug/deps/mem_layout_test-972dfe32100b3945)

running 4 tests
test dio::buffer_aligned::tests::test_bytes_put ... ok
test dio::buffer_aligned::tests::test_bytes_put_panic - should panic ... ok
test dio::buffer_aligned::tests::test_io_buf_put_slice ... ok
test dio::buffer_aligned::tests::test_with_capacity_aligned ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.51s

Checklist before requesting a review

  • [x] I have performed a self-review of my code.
  • [ ] If it is a core feature, I have added thorough tests.
  • [ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • [ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • [ ] Do not forget to reformat commit message to not include the above checklist

yliang412 avatar Aug 15 '24 03:08 yliang412