pipeline HTTP tracker requests over the same connection

[ghost]

libtorrent version (or branch): 1.2.4

please describe what symptom you see, what you would expect to see instead and how to reproduce it.

I have around 2500 torrents in qBt. If I start them all at once, the announces go through all at once and some 500-600 would fail(timed out). I think the tracker gets overwhelmed by the sudden influx of simultaneous announce requests. There should be a limit to max simultaneous announce requests. Maybe make a queue system which would send max "x" amount of announces/second.

[arvidn]

it would be even better to coalesce announces to the same tracker (as long as it's HTTP) and use the same connection.

In your case, when the trackers time out, they are re-tried pretty soon, right? Say, within 10 seconds or so.

Are any torrents left stuck not being able to announce, or do they all recover eventually? If so, does it take unreasonably long to recover?

[arvidn]

I don't think this would be characterized as "spam" though. As far as I can tell from your description, all of those announces are legitimate, right?

[ghost]

Yes they get retried. But I cannot say the same for stop event. When I stop all torrents some still get stuck at error timed out state.

[ghost]

Interesting thing, I moved everything over to rtorrent and it doesn't fail on any announce (start or stop). Maybe it handles the announces in a different way. I tried with deluge it has same problem(specially at stop event).

[arvidn]

are all or most of the announces going to the same tracker? wireshark should be able to tell you whether announces are coalesced into a single connection.

[ghost]

Ok I just found that rtorrent has this variable ‘network.http.max_open.set’ which basically limits the “maximum number of simultaneous HTTP request (used by announce or scrape requests). Default: 32“ -according to their documentation. Mine was set to 50.

So rtorrent wasn’t making 2500 announce request at once whereas libtorrent was probably making that many. Which made the tracker time out like a DoS attack.

[ghost]

Arvidn do you plan to fix this issue in some future release? I see two possible fixes: Limit the max simultaneous queries(to same/all trackers). Multiplexing all queries made simultaneously to one tracker through one tcp connection. This also implies, reusing the same TLS handshake if tracker is using SSL.

[arvidn]

I'm hoping to do the latter. I'm pretty busy with other things right now though. Would you be interested in giving it a stab? (it's in tracker_manager.cpp and http_tracker_connection.cpp)

[ghost]

The latter has some issues though. It won’t be ideal for networks with packet loss. Since you’ll be using only one TCP connection, if it faces packet loss you’d have to resend the massive announce request again. Whereas when you use multiple simultaneous connections, a few of them will get packet loss and others won’t. So you just resend those failed announces only.

[arvidn]

I don't think that is a correct characterization of how TCP congestion control works. If a TCP segment is lost, that segment needs to be re-sent, nothing else. Everyone implements Selective ACK these days, so the subsequent segments are unlikely to be resent.

You would also gain throughput by using a TCP connection whose congestion window has already been increased; i.e. you don't need to wait for slow-start again, for every announce.

The announce request is not massive, in fact, it's quite small compared to the SSL handshake. So, even if you have (non-congestive) packet loss, I would think you would have a huge gain in bandwidth, CPU usage and latency by announcing over the same connection.

[arvidn]

looking into this some more, unfortunately this is not as simple as one might think. Mostly because of the current abstraction over making HTTP requests (using http_connection). I think the best way to support this well would be for http_tracker_connection to not use the http_connection class, but instead connect directly, and use boost.beast for the HTTP protocol. I think this would probably simplify a lot of the code as well. However, it also means it's a fairly substantial patch.

The alternative would be to add yet another patch to http_connection, which is already patched beyond what I'm comfortable with. It has a lot of hacks specifically to support the http_tracker_connection.

The only other user of http_connection is the UPnP mapper.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[FranciscoPombal]

not stale

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[FranciscoPombal]

not stale

[engstrom]

@arvidn Was this was unintentionally closed or was a decision made not to implement this?

[thesyntaxslinger]

Can this be re-opened?

[iconoclasthero]

Is #7526 a more general feature request for multiplexing connections that would address the issues raised herein? I think that there's some interest in seeing this become a reality for some popular torrent programs that rely on libtorrent-rasterbar, e.g. deluge and qbittorrent.