Page-Align Writes

[eatnumber1]

Hi there,

I've been digging a lot into performance issues I experience when using qBittorrent with its disk mounted over SMB/CIFS with a high-latency network. Also relevant is that it's an asymmetric network: I can write files about 25x faster than I can read them.

What I've found is that while (when CIFS is configured correctly), the page cache can elide most writes, some writes actually produce a synchronous read. This synchronous read, when downloading a torrent, ends up being a significant bottleneck (due to the asymmetric network).

libtorrent seems to switch from using mmap to using read/write when it detects that there's a network filesystem, however this doesn't fix all of the performance issues with it.

In particular, what I believe happens is as follows:

• Download a torrent that contains multiple files, where at least one of the files in the torrent itself starts at a byte offset that is not a multiple of the page size.
• libtorrent downloads the first file. Since that file started at chunk 0 offset 0, all the chunks downloaded for that file (except the last) produce writes that are page aligned in the file on the filesystem.
• libtorrent downloads the second file (and the last chunk of the first, containing parts of both the first and the second files). Since now this file in the filesystem is not aligned with the chunks being downloaded, libtorrent starts doing page unaligned writes.
• When libtorrent does a partial write to a page (which it does at least once, maybe twice (first+last page of the write), on every write call when making unaligned writes), Linux must backfill the unwritten data from disk in order to store that newly dirtied page in the page cache.
  • This "dirty page backfill" read is synchronous, and in my case must go over the network. This happens even if flushing the dirty page is deferred, so in my case:
• After backfilling the page and marking it dirty in memory, the write call returns, having not written anything! The page will get written to the SMB server asynchronously later.

I'll use an example: if a torrent has a 1M chunk size, is exactly 3 chunks in length, and contains two files, the first file is 1M+2k (1026 KiB), and the second is 2M-2k (2046 KiB), with a 4k page size. Let's also say that libtorrent is making write calls by passing in 1 MiB buffers. You'll get:

1. download chunk 1 (file 0)
   • write 256 4k pages out to file 0 on disk
2. download chunk 3 (file 1)
   • the chunks are unaligned now with the pages. chunk 3 in file 1 on disk starts at byte offset 1M+2k (1026 KiB), which is not a multiple of 4 KiB
   • chunk 3 starts writing to file 1 at offset 1022 KiB, which is a partial write of page 256 of the file. This doesn't cause a backfill because this page is new in the file (it's an empty file)
3. download chunk 2 (spans both files 0+1)
   • write 2k into the end of file 0. As before, this does not trigger a backfill since there's no data after the end of the file
   • write the remaining 1022 KiB (255.5 pages) to the beginning of file 1
     • 255 of those pages write without backfill. The last page however (page 256) needs to backfill the data written from chunk 3 above, causing a synchronous read to disk

I think the way to fix this is by adding a small buffer to defer partial writes to pages, and if you want to get complicated/fancy with it, you could calculate "these chunks combined together allow aligned writes" and prefer downloading those together.

[Seeker2]

This seems familiar to something written here about multi-file torrents: https://forum.qbittorrent.org/viewtopic.php?p=12725#p12725

[arvidn]

libtorrent seems to switch from using mmap to using read/write when it detects that there's a network filesystem, however this doesn't fix all of the performance issues with it.

It doesn't. I suspect it's really the memory mapped file that's causing issues.

[eatnumber1]

I'm seeing it using read/write for the files via strace, so I don't think mmap is an issue.

[arvidn]

right, there's a special case for small files. They are read and written using regular read() and write() calls.