mu icon indicating copy to clipboard operation
mu copied to clipboard

Improve indexing performance: batch stat()s with io_uring

Open dcolascione opened this issue 10 months ago • 1 comments

Add support for using io_uring to batch stat() calls when scanning maildir directories. This approach reduces the number of individual syscalls by processing stats in batches of up to 16384 files at a time; it also allows the kernel to continue doing kernel-internal stat()s while we do indexing work.

Performance impact is moderate but noticeable on a real-world maildir with ~490k messages:

Without io_uring: 17.5s real time (0.75s user, 5.6s sys) With io_uring: 15.4s real time (0.74s user, 8.4s sys)

The higher sys time with io_uring reflects the batch processing happening in kernel space rather than repeated userspace->kernel transitions.

Only enable io_uring for maildir directories since:

  1. They're the only directories likely to be large enough to benefit
  2. This ensures the io_uring instance isn't used concurrently

The feature can be disabled at runtime via MU_DISABLE_IO_URING=1 or at build time via -Diouring=disabled. Requires liburing >= 2.3.

dcolascione avatar Feb 22 '25 20:02 dcolascione

Looks interesting. Will take a bit longer to review this (I'm not too familiar with io_uring), I don't think before 1.12.9 (which should be in the coming week)

The performance improvements are modest, so we'll have to weigh that against the added complexity. Anyway, thanks for doing this!

djcb avatar Feb 23 '25 09:02 djcb