firecracker
firecracker copied to clipboard
firecracker-microvm
[Perf] Implement async IO for the block device using io_uring
After some more debugging there is no doubt that there are 2 major issues which bring down the overall performance of our block device emulation:
- serialized block file I/O
- high guest interrupt rate
Our block device emulation works by sequentially processing guest I/O requests. It will also serialize the latency of each request and this results in very low CPU (host/guest) usage when running fio benchmarks. The Firecracker emulation thread is mostly sleeping/waiting for I/O to complete and fio maxes out 4-5K IOPS. To improve on this we will need to add block async I/O support in Firecracker.
The increased guest interrupt overhead becomes significant only when the Firecracker block device is doing a lot of IOPS (70-80k). That happens only if it is backed by a RAMdisk to minimize latencies. As an improvement we will need to implement virtio driver and device event suppression / interrupt mitigation. I've experimented a bit with this and got a nice performance boost.
The current plan is to use io_uring to parallelize the block I/O operations. The io_uring interface was added in kernel 5.1 so this will be available only when running on host kernels that support io_uring, otherwise the emulation will work serially as before.
We're starting to prototype this out now (we'll post some details here once we have some outcomes).
This was released as dev preview in Firecracker v1.0.0 More info in the docs: https://github.com/firecracker-microvm/firecracker/blob/main/docs/api_requests/block-io-engine.md
We'll close this issue once we make the Async block io_engine GA.
Moved to shipped because released with Firecracker v1.0.0 and not further development is required.
Sorry for moving the issue again, moving it back to dev preview because the condition to make such feature GA listed here are not met yet.
