qBittorrent
qBittorrent copied to clipboard
qbittorrent
File checking speed for 4.4.0 is much slower than 4.3.9
qBittorrent & operating system versions
qBittorrent: 4.3.9 x64 Operating system: Windows server 2022 datacenter 21h1 libtorrent-rasterbar: 1.2.14 qt: 5.15.2
What is the problem?
qbittorrent v4.4.0 with qt6 file check speed is extremely slow, compare to qbittorrent v4.3.9. About 2x slower for hdd, or 10x slower for samba protocol. require for a fix, thanks!
Steps to reproduce
No response
Additional context
No response
Log(s) & preferences file(s)
No response
If slower rechecking speed is only with QT6.2.2 and qBitorrent 4.4.0 combination and QT5 build of 4.3.9 is faster, then the beginning of your issue needs to be edited, because otherwise it will confuse people to think your issue is observed on old version.
For fair comparison this should also be tested with QT5 + qBit 4.4.0 combination and all other torrents should be put to pause and torrent data should be on another drive not where is your Windows, then hopefully there's no extra background activity slowing down drive read with torrent data and then can be more sure if QT6 has a bug as you thought or if the blame leans towards the usage of Libtorrent 2.0.5 or something else slowing down CPU or drive read operations.
Would be also good to know the total file size, how many files in torrent and how long does each version recheck.
Same issue here, I made no changes other than upgrading program version. 4.3.9 checks files at 400MB/s+ I switch to 4.4.0 check same file, speeds drop to 100MB/s. Reinstall 4.3.9, check file, back to 400MB/s+,
Slow check on 4.4.0 and hangs/freeze hardly (feeling of 1~5 FPS on UI) Downgraded to 4.3.9 and everything is fine
Windows 10 x64 21H2 Build - 19044.1466
I'm not sure if this is the cause, but I was just doing a force recheck on a 38GB torrent, and I have 32GB of RAM. I noticed my client became very sluggish and looked over to my task manager to see my memory usage was at 100%. Stopping the recheck completely(just a pause didn't work) and my memory was freed. When I tried again I could see my memory usage climbing as the recheck progressed. Restarting qBittorrent didn't fix the issue and memory usage again kept climbing trying to recheck the torrent. Windows 10 qBittorrent: 4.4.0 Qt: 5.15.2 Libtorrent: 2.0.5.0 Boost: 1.78.0 OpenSSL: 1.1.1m zlib: 1.2.11
Same issue here. All I did was update to 4.4.0 and the file recheck is extremely slow now. There may be a memory leak, several times my ram maxed out and qBittorent was using it all. Win 10 with 16Gb ram, qBittorrent was using using 9Gb at one time during file recheck. This started when I noticed a torrent wasn't updating file progress during download. It downloaded over 100Gb of a 150Gb file and showed about 9% done, initiated file recheck and it has been running for 2 hours trying to recheck.
Is qBittorrent v4.4.0 with Libtorrent 1.2.15 still slower than qBitorrent v4.3.9?
Is qBittorrent v4.4.0 with Libtorrent 1.2.15 still slower than qBitorrent v4.3.9?
Which can be found here on the bottom https://github.com/qbittorrent/qBittorrent/actions/runs/1665307409 It will probably be back to normal with this version because rechecking is a libtorrent operation.
The GUI keeps freezing for 2 to 3 seconds with 4.4.0. Reverting to 4.3.9 has zero freezing. Re-loading 4.4.0 again causes constant freezing.
just verified a 12GB torrent. the hard disk usage was very bad. hdd sounded bad, run at 100% and was doing only reading at very low speed. everything related to that drive was disturbed. 16gb ram almost filled by the process.
...after some testing, reverted to 4.3.9, all the semi-verified torrents verified normally, while seeding multiple torrents. no ram bloating. hdd works and sounds normally, did not even got to 100%, id guess.
Seeing the same behavior on my system as well on 4.4.1. When checking a torrent, it was maxing out at 7 MB/s. Once I rolled back to 4.3.9 (keeping the same configuration), it's now checking at 220-230 MB/s.
Seeing the same behavior on my system as well on 4.4.1. When checking a torrent, it was maxing out at 7 MB/s. Once I rolled back to 4.3.9 (keeping the same configuration), it's now checking at 220-230 MB/s.
Same on 4.4.2 rolled back to 4.3.9
Can anyone confirm that using latest qBittorrent v4.4.2 with Libtorrent 1.2.15 solves the problem and that issue is still with default Libtorrent 2.0.5+?
Doing the test with Libtorrent version change was recommended in January, but no one here still hasn't confirmed if it solves anything.
Official website has written that now there's official installers made available with Libtorrent 1.2.15+, so no need to downloading GitHub .zip builds in actions page to do tests.
https://www.qbittorrent.org/download.php Qt5 and Qt6 versions using libtorrent 1.2.x (RC_1_2 branch). Only 64bit versions. These are offered to help the transition from v4.3.x to v4.4.x and to allow users to test for possible regressions. They may not be offered for future versions.
im done updating, until this debacle is long gone. i read other issues and the developers seem to have no clue, of what they are doing.
Also stumbled upon this issue. Was able to reproduce it with 4.4.0, 4.4.1, 4.4.2 versions (libtorrent 2.x) Changing the number of hashing threads did not bring any results.
How can I help debug this problem?
Also stumbled upon this issue. Was able to reproduce it with 4.4.0, 4.4.1, 4.4.2 versions (libtorrent 2.x) Changing the number of hashing threads did not bring any results.
How can I help debug this problem?
I've test 4.3.9~4.4.2, I can confirm it is OK with 4.3.9, but quite slow with other versions.
Compiled qbittorrent-nox from master in two flavors — with libtorrent 1.x and 2.x
Tested on 25G torrent. 1.x version takes ≈7s to rehash, 2.x takes 12-13s. Looks like a problem with libtorrent, not qBittorrent.
@PriitUring
Orhideous, it would be nice, if you do the same test on a good old hardware, like what we, mortals use :). such tests make any flaws quite obvious.
25GB in 7~12sec is quite fast, at least now we know that the speed difference between libtorrent versions with different hardware or OS setup and latest qBittorrent v4.5.0alpha1 isn't capped at low 400mb/s vs 100mb/s as some long ago had experienced, yet they then had not shared much other extra information about how exactly tests where performed to be able to reproduce by others.
@Orhideous
- What CPU?
- How much RAM?
- What kind of drive is the torrent saved at?
- Is your OS on different drive?
- Is the Torrent file a single 25GB file or multiple files?
- Did you only had one torrent kept in your list?
- Did try reseting your settings?
- Does it help at all if you use latest Masters default settings and increase your qBittorrent physical RAM limit from default very low 512 MiB to be changed to ~70% of your RAM?
- Did you restart system between your various tests?
- How many secs does the recheck take with latest stable qBittorrent version with different libtorrents vs the Master branch builds?
I just update to 4.4.2 from 4.3.9, obviously 4.4.2 file checking speed is much more slower than 4.3.9, the files checked are saved on SSD. So disappointing that 4.4.x are still so much bugs, i am thinking whether should i roll back to 4.3.9.
@PriitUring Here is some infomation: OS: Windows 11 Pro 21H2 x64 (22000.613) RAM: 16GB*2 + 8GB*1 Files: A folder contains 54 files, each one is 1.x GB and total are 72.4 GB. Drive Type: SSD
qbittorrent_4.3.9_x64_setup.exe + libtorrent 1.2.14.0 (Default settings): 700+MB/s qbittorrent_4.4.2_x64_setup.exe + libtorrent 2.0.5.0 + Physical memory (RAM) usage limit: 512MB (Default Value): 310+MB/s qbittorrent_4.4.2_x64_setup.exe + libtorrent 2.0.5.0 + Physical memory (RAM) usage limit: 16384MB : 380+MB/s qbittorrent_4.4.2_x64_setup.exe + libtorrent 2.0.5.0 + Physical memory (RAM) usage limit: 32768MB : 380+MB/s qbittorrent_4.4.2_RC_1_2_qt5_x64_setup.exe + libtorrent 1.2.15.0 + Physical memory (RAM) usage limit: 512MB (Default Value): 700+MB/s
I think libtorrent 2.x is the cause of the problem.
@microka Please make a backup of your torrents resume data and settings folder, then try testing these qBittorrent versions with default settings and only 1 single torrent:
- v4.3.9 vs v4.4.2 vs latest v4.4.x branch build with Qt5 and libtorrent 1.2.15+
- v4.3.9 with Qt5 and libtorrent 1.2.14+ vs v4.4.2 and latest v4.4.x with Qt5 and libtorrent 2.0.5+
- same speed tests with v4.4.2 and latest GitHub v4.4.x branch build of qBittorrent with libtorrent 2.0.5+ and only modified setting afterwards is RAM limit increased higher than low 512 MiB.
Also specify what OS, ram amount, storage drive type you're using and torrent file size.
Because maybe some bugs are already fixed in newer v4.4.x GitHub build or just some dependency, outdated or unsuitable setting is causing you various bugs and maybe latest qBittorrent version isn't really any worse if you just reset settings to defaults or change something.
Without doing simple various fair equal tests or mentioning what additional new major qBittorrent version bugs still needs to be fixed and really are not seen in older version is just not otherwise not helpful.
The tests i ran above is extract the setup .exe file using 7-zip, and create a 'profile' folder to run qBittorrent with default settings in portable mode, each one test only 1 single torrent.
-
v4.3.9 (Qt 5.15.2 + libtorrent 1.2.14.0) vs v4.4.2 (Qt 5.15.2 + libtorrent 2.0.5.0), the two test results I'have already posted. But i don't know how to run the 'latest v4.4.x branch build with Qt5 and libtorrent 1.2.15+' in portable mode.
-
v4.3.9 with Qt5 and libtorrent 1.2.14+ vs v4.4.2 and latest v4.4.x with Qt5 and libtorrent 2.0.5+ except latest v4.4.x, other results already posted.
-
I don't know how to get "latest GitHub v4.4.x branch build of qBittorrent with libtorrent 2.0.5+"
o bloody hell..so this is the cause? my hdd was freaking out , and omg the sounds it was producing , like someone is ripping it out of skin alive i also waited 40 minutes and it only went 15% checking.. reverted back to 4.39 and while i was typing this 15 seconds? its already at 50%.... and no sound is heard while doing it at all win10 x64 WD Black
Easiest fix: Download 4.3.9 and disable 'Check for program updates' ¯(°_o)/¯
Quite slow. 400 torrents will cost 2 days. But for v4.3.9, several hours are enough.
do the libtorrent-1.2.x and libtorrent-2.0.x tests use the same number of threads to hash with? (this is the aio_threads setting).
If it's hard to tell, when looking at activity monitor, does one peg more CPU cores than the other?
so maybe fewer aio_threads would be better in 2.0.x. it would probably increase "sequentiality"
Problem
«New versions of qBittorrent are very slow in checking files!» «It's broken, fix it!» «The developers themselves don't know what's going on...» «Don't use the new versions.»
Enough.
There have been a lot of complaints like this. It's time to investigate and fix this problem. Rolling back to old versions is not a good option, because turning a blind eye won't make the bug go anywhere. As much as one might wish that to happen.
We need to figure out if there is a problem at all, if there is, which component in «libtorrent-qBittorrent-hardware» chain is causing it, and how to solve it.
So... test time!
Hardware used
- CPU: AMD Ryzen 9 3900X 12-Core 4.8GHz per core
- RAM: 64G KHX3600C18D4
- NVMe: WD Black SN750 NVMe PCIe M.2 250GB
- HDD: HGST 5K1000-750 SATAII 5400rpm
Software used
- Host: Kubuntu 21.10 (impish)
- Kernel: 5.13.0-40-generic
- Virtual machines: Kubuntu 22.04 LTS (jammy), 8G RAM, 1/4/12 cores.
- VirtualBox: 6.1.32-dfsg-1~ubuntu1.21.10.1
Testing methodology
I installed Kubuntu 22.04 LTS in a virtual machine, installed all the build dependencies, and then cloned the base of that image into three instances.
- In the first (qBt_1) - from the
masterbranch (commit 9351f66c) compiled qBittorrent 4.5.0alpha1 with libtorrent-rasterbar 1.2.16 - The second one (qBt_2) - from the
masterbranch (commit 9351f66c) compiled qBittorrent 4.5.0alpha1 with libtorrent-rasterbar 2.0.6 - The third one (qBt_0) - from tag release-4.3.9 (commit 01519b5e) compiled qBittorrent 4.3.9 with libtorrent-rasterbar 1.2.16
Compilation instructions are taken from CI workflow, GUI=ON version was compiled.
A 32G payload.bin file was created from /dev/urandom as test data. This file was placed on two 40G virtual disks, NVMe and HDD; v1 torrent was created for the file, 16384 parts of 2M.
Before each test time measurement, the caches were reset both on the host and in the virtual machine, echo 3 > /proc/sys/vm/drop_caches
Each value was obtained by averaging 4 measurements. For the clients, all settings are left default except two, aio_threads and hashing_threads.
Numbers in tables below represents seconds taken to recheck single torrent, if not specified otherwise.
Results
First, the base values on an «old good qBittorrent» in qBt_0, i.e. 4.3.9, with all default settings.
Baseline
4.3.9, aio_threads=10
| Cores | HDD | NVMe |
|---|---|---|
| 1 | 319±2 | 57±1 |
| 4 | 320±2 | 32±1 |
| 12 | 358±3 | 31±1 |
Broken qBittorrent?…
Let's test the first hypothesis that the qBittorrent update caused the slow scanning in some way. To do this, let's compare 4.3.9 and 4.5.0 on the same version of libtorrent-rasterbar, 1.2.16
HDD, 4.3.9 vs 4.5.0, libtorrent-rasterbar 1.2.16, aio_threads=10
| Cores | 4.3.9 | 4.5.0 | 4.5.0 (aio_threads=1) |
|---|---|---|---|
| 1 | 319±2 | 325±3 | 322±2 |
| 4 | 320±2 | 325±2 | 322±2 |
| 12 | 358±3 | 358±1 | — |
NVMe, 4.3.9 vs 4.5.0, libtorrent-rasterbar 1.2.16, aio_threads=10
| Cores | 4.3.9 | 4.5.0 | 4.5.0 (aio_threads=1) |
|---|---|---|---|
| 1 | 57±1 | 60±1 | 60±2 |
| 4 | 32±2 | 58±1 | 57±1 |
| 12 | 31±1 | 33±1 | — |
According to the results of the tests, there is no difference in performance between the versions of qBittorrent itself.
Problems with libtorrent?
Let's check the second hypothesis: maybe the performance problems are caused by some regression in libtorrent-rasterbar.
4.5.0, libtorrent-rasterbar 1.2.16 vs 2.0.6, NVMe
aio means aio_threads, hashing means hashing_threads
| Cores | 1.x aio=1 |
1.x aio=10 |
2.x aio=1 hashing=1 |
2.x aio=1 hashing=10 |
|---|---|---|---|---|
| 1 | 60±1 | 60±1 | 60±1 | 59±1 |
| 4 | 57±2 | 58±1 | 53±2 | 32±2 |
| 12 | — | 33±1 | — | 22±1 |
It turned out that libtorrent had nothing to do with it either. But wait a minute…
Problems with specific storage medium?
Let's test it again — but with spinning rust this time!
4.5.0, libtorrent-rasterbar 1.2.16 vs 2.0.6, HDD
| Cores | 1 | 4 |
|---|---|---|
1.x aio=1 |
322±2 | 322±2 |
1.x aio=10 |
325±3 | 325±2 |
2.x aio=1 hashing=1 |
337±1 | 322±1 |
2.x aio=1 hashing=2 |
496±3 | 390±1 |
2.x aio=1 hashing=4 |
807±1 | 804±2 |
2.x aio=1 hashing=10 |
916±3 | 868±1 |
Finally.
Conclusions
IOPS starvation, that's it. In SSD-baked setups with plenty of IOPS the problem does not arise at all. The situation changes dramatically on HDDs, especially consumer range HDDs, especially with data fragmentation. Once upon a time @arvidn wrote a note:
It seems HDD performance drops significantly at > 2 hasher threads, whereas SSD performance (which is most likely CPU bound for the most part) just improves the more threads thrown at it.
That's exactly what's happening. Performance significantly drops even at 2 hasher threads. In short, this is a fairly obvious thing described in libtorrent's documentation.
The hasher threads do not only compute hashes, but also perform the read from disk. On storage optimal for sequential access, such as hard drives, this setting should probably be set to 1.
So what to do?
Set Hashing threads value to 1.
Set Hashing threads value to 1 unless you are absolutely sure you know what you are doing! Values greater than 1 can increase performance only in two cases:
- full SSD setup;
- RAID0 or similar topology (for example, raidzN×N in ZFS) capable of sufficient IOPS.
I propose to change the default number of hashing threads to 1, instead of 2, as was done in #13499. It may be worth migrating in the configuration and forcing this value to be overwritten, so that at least the user experience is not degraded. /cc @FranciscoPombal @bershanskiy
Useful links
- arvidn/libtorrent#4734
- #13499
- Disk I/O settings for checking files
- hashing_threads
interesting post, but im not convinced, that you have found the issue. a lot of code has been changed in libtorrent 2. and hdds behave abnormally, ist not about time-to-finish-a-job. that is expected behavior, as you say yourself.
@linax3-genuine can you describe the issues you experience when you set hashing_threads to 1?
@Orhideous could you test this as well by setting the hashing threads to zero and see the outcome? https://github.com/qbittorrent/qBittorrent/pull/16956