rakshasa
Request for testing and recommendation of config presets
I'm planning on adding various config presets to make it easier for users to optimize their client.
This will mainly be related to SSD/HDD storage for torrent data and session files (e.g. HDD for torrent data, SSD for session data), high/normal bandwidth, seeding very large number of torrent's vs. relatively normal number doing both leeching/seeding (need to disable fdatasync on session files), etc.
A new command called config.preset will be added that will take a list of string arguments.
E.g. default, default.ssd, default.hdd_data.ssd_session, seeder.hdd, etc.
So please post below recommendations for what these options should be, preferably backed by real-world data.
I run rtorrent on a NAS (mid-level 12th gen i5, 64gb RAM, metadata on SSD, torrent data on an 8 spindle HDD array, gigabit WAN connection)
I seed about 1000 torrents: the average size is around 50-60GB, and 5-10 are active at any given time. Upload is capped at 250KB/s, and is usually at the limit. Download is uncapped (gigabit line speed) it generally isn't more than 5-10 active torrents at a time.
My config probably isn't optimal, but rtorrent doesn't obviously misbehave and I get the throughput I expect:
pieces.memory.max.set = 8000M
network.http.max_open.set = 64
network.max_open_files.set = 32768
network.max_open_sockets.set = 2048
throttle.max_uploads.set = 100
throttle.max_uploads.global.set = 500
throttle.max_downloads.set = 100
throttle.max_downloads.global.set = 500
pieces.hash.on_completion.set = no
encoding.add = utf8
pieces.preload.type.set = 0
network.receive_buffer.size.set = 16M
network.send_buffer.size.set = 16M
network.tos.set = throughput
pieces.memory.max.set is key. I originally had this as 1200 in an old config and was experiencing strange 'ramping' in connection speeds (download speed would increase, hit a specific rate, drop to near-zero, and repeat) and could not reach or sustain gigabit. Through experimentation I found that 3600 was enough to get the ramping to stop, but set it to 8000 for margin.
Hopefully that's helpful.
I run an second hand HP z240 SFF with 32GB ram and 2x1TB ssd, around 500Mbps wan. Mainly used as all i need home helper machine.
I use rtorrent more as a normal download client without seeding a massive ammount of torrents. What is downloaded is later deleted when a good seed ratio is reached. Nothing crazy just normal stuff.
Hope this helps.
Given the variety of performance that different host configurations can provide, (eg, raspberry pi 2 vs HP Proliant 380 G11 384GB Ram, Synology UC3200 SAN with full SSD dual controllers, on an iSCSI dedicated network. All in 10Gbps) can I suggest a performance test widget that provides guidance on various config elements rather than dedicated predetermined configs?
That way you can avoid the temptation people face to make everything faster, and can possibly guide people to more balanced systems, where the performance across bandwidth, latency and capacity are aligned based on measured performance on the system in use.
A new command such as config.test could read the current config and report the performance stats of the configured storages, or alternatively, when given the applicable command line variables, provide suggested configuration changes.
For file counts, perhaps, 100 files, 500 files, 1000, 2500, 5000 file configs, and for file sizes with either average file sizes given or maximum file sizes given, then the appropriate config option recommendations.
If the internet bandwidth is 25Mb, 100Mb, 250Mb, or 1Gb, then provide appropriate configuration recommendations for 25%, 50% 75% and 100% use of the bandwidth. (I don't think that labels such as 'high' or 'normal' for internet bandwidth are helpful given the range of bandwidth speeds available).
Sabnzbd has a nice little interface widget that reports a variety of system stats.
System performance (Pystone) 451981 12th Gen Intel(R) Core(TM) i5-12600H AVX2
Download folder speed 112.7 MB/s /mnt/sab_incomplete
Complete folder speed 19.6 MB/s /media/truenas/Hosts/sabnzbd/
Internet Bandwidth 1.44 MB/s 11.52 Mbps
Platform lxc Ubuntu 22.04.5 LTS
I'm mostly interested in the following options, which are for the SSD/HDD variants:
system.files.advise_random.set
system.files.advise_random.hashing.set (to be implemented)
pieces.preload.type.set
pieces.preload.min_size.set
pieces.preload.min_rate.set
The more optimized or specific speeds, bandwidths, etc, would be for later. Memory, open sockets and such is a close second to the above.
Those could be configs written as method that take the up/down speeds and optimizes the throttle, etc, settings.
I'm still on 0.15.1 because of the crashes so I don't think I have advise_random yet.
pieces.preload.type.set=2
pieces.preload.min_size.set=50000
As suggested by stickz in a different issue might have had some positive improvement, but it's hard to tell due to the inherent randomness of BT... do you have any suggestions either for simulating load, or for measurement?
system.files.advise_random.hashing.setis now available.
It's commented out in 0.15.3
https://github.com/rakshasa/rtorrent/blob/v0.15.3/src/command_local.cc#L225-L226
Finally got around to adding system.files.advise_random.hashing.set, so much appreciated if people benchmark it.
Can this change be reflected in the documentation with some examples? Are system.files.advise_random just true|false kind of options?
-
system.files.advise_random.set = false(default): Do not advise files and memory mapped chunks, so the kernel uses the default which is most likely some for of sequential read-ahead. -
system.files.advise_random.set = true: Advise POSIX_FADV_RANDOM and MADV_RANDOM. -
system.files.advise_random.hashing.set = false(default): Do not advise files and memory mapped chunks during full torrent hash checking. This does not affect files or chunks hashed during download. -
system.files.advise_random.hashing.set = true: Advise POSIX_FADV_RANDOM and MADV_RANDOM during full torrent hash checking.
Are there any recommendations of system.files.advise_random.set and system.files.advise_random.hashing.set for HDD or SSD?
HDD: false, false SSD: true, false
Seems the right ones, however I haven't properly tested them.
Both SSD and HDD should be true, false.
Hash checking is sequential regardless of if it's coming from SSD or HDD, and seeding is random, regardless of if it's coming from SSD or HDD.
Hi
What's best setting for pieces.preload.type.set at the moment? madvise?
What's best setting for pieces.preload.type.set at the moment? madvise?
Madvise, definitely don't want direct paging as that is only a hack to deal with systems where madvise isn't available. In fact I should probably remove that option.
@rakshasa I'd request/recommend that you leave direct paging, as there are many cases where madvise likely isn't available in 2025.
For example - On Linux with NFS/CIFS mounts, ZFS filesystems, etc.
In these cases, madvise will likely just silently fail.
It's been a while since I actually checked what is happening under the hood for these, but, historically, madvise's readahead was only activated for traditional local filesystems like ext3/ext4. ZFS was excluded because it has it's own file caching system, and CIFS/NFS were excluded because of it potentially being a bad idea over a network ...
But in the case of rTorrent, if the user wants to force it, i'd say give them the option -- and it's not like this is a 1000 line-of-code feature. Especially when you consider that NFS, CIFS, and ZFS are all likely commonly used for rTorrent users.
I am currently running on a Raspberry Pi (via crazymax Docker image), torrent metadata on the Raspberry Pi NVMe drive, and torrent download folder on a NAS within local network (HDD). ISP connection is 8 Gbps, on 2.5 Gbe local ethernet network.
The following gets me 3.5 mo/sec (28 mbps) maximum during download
system.files.advise_random.set = true
system.files.advise_random.hashing.set = false
The following gets me ˜40 mo/sec (400 mbps) maximum during download
system.files.advise_random.set = false
system.files.advise_random.hashing.set = false
I haven't measured hashing speed.
Not scientific measurements at all, but clearly a difference for HDD setup with the latest parameters.
I run an second hand HP z240 SFF with 32GB ram and 2x1TB ssd, around 500Mbps wan. Mainly used as all i need home helper machine.
I use rtorrent more as a normal download client without seeding a massive ammount of torrents. What is downloaded is later deleted when a good seed ratio is reached. Nothing crazy just normal stuff.
Hope this helps.
Thanks @mirolm i was looking so long time for fast seed. rtorrent configs really hard to do for beginners
I will apply this with some changes for my 1 gb ram 1 core cpu 1 gbps (practically i take 500 mbps arounds) 60 gb storage server. With close to default config my server was seeding and i was trying download it with my phone (around 70 mbps). Server was giving very low speeds like 600 kbps around, while there is no load on server side. Only 1 torrent and very low internet usage. And the torrent's seed not much widely seeded. So i was able to see my server seed was bad with increase/decrease of download on my phone.
Though i was using debian so my rtorrent version was 0.9.8 the reason of low speed seeding may be also this.
@rakshasa I'd request/recommend that you leave direct paging, as there are many cases where madvise likely isn't available in 2025.
For example - On Linux with NFS/CIFS mounts, ZFS filesystems, etc.
In these cases, madvise will likely just silently fail.
It's been a while since I actually checked what is happening under the hood for these, but, historically, madvise's readahead was only activated for traditional local filesystems like ext3/ext4. ZFS was excluded because it has it's own file caching system
This is still (partially) the case for ZFS: https://github.com/openzfs/zfs/blob/d147ed7d2655c3a3499a9f3e0f76ba66bf7face1/module/os/linux/zfs/zpl_file.c#L705-L733
POSIX_FADVISE_SEQUENTIAL has an effect, POSIX_FADVISE_RANDOM is silently ignored.
madvise/posix_madvise are more complicated because of the weird interaction in mmapped files between zfs's internal caching + the linux page cache... and I think the effect depends on your version of openzfs.
@rakshasa I'd request/recommend that you leave direct paging, as there are many cases where madvise likely isn't available in 2025. For example - On Linux with NFS/CIFS mounts, ZFS filesystems, etc. In these cases, madvise will likely just silently fail. It's been a while since I actually checked what is happening under the hood for these, but, historically, madvise's readahead was only activated for traditional local filesystems like ext3/ext4. ZFS was excluded because it has it's own file caching system
This is still (partially) the case for ZFS: https://github.com/openzfs/zfs/blob/d147ed7d2655c3a3499a9f3e0f76ba66bf7face1/module/os/linux/zfs/zpl_file.c#L705-L733
POSIX_FADVISE_SEQUENTIAL has an effect, POSIX_FADVISE_RANDOM is silently ignored.
madvise/posix_madvise are more complicated because of the weird interaction in mmapped files between zfs's internal caching + the linux page cache... and I think the effect depends on your version of openzfs.
Is there a way to determinate if I use madvise or direct paging? i'm running Ubuntu on EXT4 and on the second one I have ZFS. Are there any recommendations for both?