sync/maindata speedup via torrent changes history and the "t10k" problem

[FranciscoPombal]

Issue created to track progress on implementing the ideas discussed on this thread: https://github.com/qbittorrent/qBittorrent/issues/10999.

Though the original issue was solved, the sync/maindata endpoint can still be significantly sped up, by implementing a "torrent history" mechanism.

The idea is that each time a sync request is made, the API backend monitors the changes in the state (essentially libtorrent alerts) until the next request. When the next request does come, only the changes need to be serialized, and can be sent right away. This contrasts with the current method, whereby the whole state is serialized each request and the differences to be sent are computed by comparing two full serialization results (the previous one and the current one) each request as well.

The speed up resulting from implementing this change could finally enable qBittorrent to become a "t10k" client (meaning a client capable of reasonably handling 10s of thousands of torrents). Today, even on powerful machines, qBittorrent is not that usable with such high numbers of torrents loaded (talking about headless/WebAPI usage here).

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[hbh7]

This would be great. I've recently tried having around ~1500 torrents and it's slowing to a crawl and almost unusable. The api request to sync/maindata frequently times out with a net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK) error, sometimes showing the red unreachable message at the bottom left.

[namazso]

I read through #10999 and I don't think those suggestions would fix my (#15067) problem. Full updates are still needed sometimes and they simply don't finish under 30 seconds. To me it sounds like full update performance needs to be fixed first, I don't think serializing some 100 kilobytes of json should take half a minute on any modern processor

[beyondmeat]

Any ideas / eta when this can be addressed. I hit 10K seeds and qbit webui is taking a huge long dump to do anything.

[glassez]

Any ideas / eta when this can be addressed.

#18394

[ftc2]

awesome, @glassez !!

i have AMD FX-8350 and a bit over 3k torrents, and when i add a few new torrents (and downloading starts happening), the webui is totally unusable until those downloads complete.

do you think it might come in 4.5.3? or is it still pretty early in testing? if you need another tester, i could try.

[thalieht]

do you think it might come in 4.5.3?

4.6.0. You can already test if you want https://github.com/qbittorrent/qBittorrent/actions/workflows/ci_windows.yaml?query=branch%3Amaster (first is latest commit).

[ftc2]

nice

ok, i will give it a whirl. i'm trying this build script just because i think it's sexy:

docker run -it -w /root -v $HOME/qbt:/root alpine:edge /bin/ash -c 'apk update && apk add bash curl && curl -sL git.io/qbstatic | bash -s all --qbittorrent-master --optimize --icu'

:)

i think it's using libtorrent v2.0.9 because i forgot to configure that. hopefully i don't need master.

[ftc2]

ok, here is my unscientific test. took me about an hour.

method

AMD FX-8350, >3k torrents seeding (different machine running browser accessing WebUI)

idle test

• set global upload limit to 3 MiB/s (not downloading anything)
• wait for steady state upload
• observe maindata latency

load test

• set global upload limit to 10 MiB/s and download limit to 0
• add 8 well-seeded torrents from public tracker
• t=0: restart qbittorrent
• open webui in fresh browser container
• t=60s: observe maindata request latency, session download xfer, session upload xfer, CPU

i repeated both tests a couple of times for each setup and got pretty consistent results.

results

qBittorrent v4.3.9
Qt: 	5.15.9
Libtorrent: 	1.2.19.0
Boost: 	1.82.0
OpenSSL: 	3.1.0
zlib: 	1.2.13.zlib-ng

idle (uploading @ 3MiB/s):

• maindata latency: ~1500–2000ms

load @ t=60s:

• maindata latency: could not determine
• session dl: could not determine
• session ul: could not determine
• CPU: 60-70%

CPU-starved to the point of the WebUI not even loading. couldn't get data.

qBittorrent v4.5.2
Qt: 	6.5.0
Libtorrent: 	1.2.19.0
Boost: 	1.82.0
OpenSSL: 	3.1.0
zlib: 	1.2.13.zlib-ng

idle (uploading @ 3MiB/s):

• maindata latency: ~700ms

load @ t=60s:

• maindata latency: typically >2000ms (5000ms or even 10000ms typical)
• session dl: 1.9GiB
• session ul: 360MiB (achieves the 10MiB/s cap by t=60s)
• CPU: 50%

qBittorrent v4.6.0alpha1
Qt: 	5.15.9
Libtorrent: 	2.0.9.0 <--- note this
Boost: 	1.82.0
OpenSSL: 	3.1.0
zlib: 	1.2.13.zlib-ng

idle (uploading @ 3MiB/s):

• maindata latency: typically <100ms (20ms not uncommon), at most 200ms if i scroll through my long torrent list and start randomly clicking, but such spikes might have been coincidence, idk

load @ t=60s:

• maindata latency: <100ms
• session dl: 1.3GiB
• session ul: 30MiB – never seems to get past 1MiB/s upload if it's downloading fast (even after several minutes!!)
• CPU: 40%

amazing web api performance. piss-poor bt performance.

qBittorrent v4.6.0alpha1
Qt: 	5.15.9
Libtorrent: 	1.2.19.0
Boost: 	1.82.0
OpenSSL: 	3.1.0
zlib: 	1.2.13.zlib-ng

idle (uploading @ 3MiB/s):

• maindata latency: <100ms

load @ t=60s:

• maindata latency: <100ms
• session dl: 1.9GiB
• session ul: 200MiB (it does eventually achieve the 10MiB/s cap)
• CPU: 55%

conclusion

bt performance

libtorrent: v1.2.19.0 >> v2.0.9.0

• not sure why, but upload was severely gimped while downloading fast. i wonder if this can be mitigated by tuning libtorrent v2's settings?

qbittorrent: 4.5.2 > 4.6

• see results

maindata webapi performance (WebUI)

qbittorrent: 4.6 >> 4.5.2 > 4.3.9

• 4.3.9 WebUI is unusable under load
• 4.5.2 is usable but crap
• 4.6 is like 1–2 (sometimes almost 3) orders of magnitude better in any scenario

btw, all versions tested take ~9s to initially load the webui

[ftc2]

this 4.6 maindata speedup is easily the biggest improvement to qbittorrent in the ~3 years i've been using it. i love it!