Implement BEP 19 - Web Seeds - HTTP/FTP Seeding

[jorgemjorge]

Using HTTP or FTP servers as seeds for BitTorrent downloads as described in BEP-19

[jorgemjorge]

I currently implementing the BEP-19. I making it as module called HTTPWebSeeds.

[atomashpolskiy]

@jorgemjorge , your effort is very welcome. This would be a really nice feature to have, please keep us informed on how it goes.

A few weeks ago I had an e-mail conversation with one guy on how to approach BEP-19 implementation in Bt. I think I will post an excerpt from it here in order to provide my overall design considerations and some Bt-related details that might be helpful:

---

Speaking of url/web seeds, this would be a nice feature to have (I'd personally vote for starting with BEP-19 HTTP, as the most straightforward and widely used). It will most certainly require a substantial effort to implement though, because it obviously does not fit very well in the simplistic design that Bt currently employs. Aside from trivial updates to metadata parser and torrent object model, the following points need to be addressed:

1. All peer connections are persistent and managed via a connection pool with periodic cleanup. Establishing a peer connection involves encryption negotiation and handshake procedure (standard and extended handshakes, and sending bitfield). For web seeds we can use persistent HTTP connections via keep-alive, but other aspects will be different, namely connection establishing (e.g. in future we may want to add support for authentication) and some configuration options (e.g. inactivity period after which the connection is closed by pool cleaner).
2. Peer connections operate on BitTorrent messages, like 'request' and 'piece'. HTTP connection implementation should handle translation of these into GET request and response.
3. Bt has 3 piece selection strategies: sequential, rarest-first and randomized rarest-first. All of these won't work well in the presence of web seeds, because of the underlying assumption that it's OK to request one piece at a time, which is untrue for HTTP servers. BEP-19 describes the necessary modifications for maintaining larger contiguous byte ranges, consisting of several pieces, which can be "assigned" to web seeds. While it's trivial to modify piece selection, it's going to be tricky to make use of this modifications in the piece assignment algorithm (which maps selected pieces to peers). The code as it is now is, quite frankly, shit, so it will most certainly require some refactoring.
4. Piece selection strategy is currently specified by the library user upon creating a torrent session (client). Given that we don't know in advance if webseeds are going to be used (as they are discovered from the metadata file), for each selection strategy its' BEP-19 version should be enabled automatically, if web seeds are present. Also, sequential strategy, that is commonly used for streaming, might be very tricky to use with web seeds.

---

To start we could keep it simple (and disregard authentication, keep-alive

I agree, but we'll still need to re-use the existing infrastructure, which currently implies that connections are long-lived, not per-request. Which makes it necessary to use keep alive (though I don't see it as a complication).

Do you have a clear idea regarding the required modifications that should be done in the core?

Rather a general understanding of the list of things that should be done. Also, I don't want the http library in the core, so everything, that can not be abstracted in such a way that http dependency is not required, should be done in a separate module (which is now called bt-http-tracker-client and will be a subject to renaming after the new bep is added). This can impose a real challenge in terms of design unless we're able to make the logic of working with webseeds really generic. Better way to put it would be: while the notion of webseeds is going to be present in the core, the different related parts of the implementation should be easily "turned off", when http module is not present in the application. In that sense what needs to be generified is not the webseeds, but rather the core API that is affected: connections/messaging and piece selection/assignments. With the latter being the "tricky" part.

[jorgemjorge]

Thanks for hints, I will take them into consideration. Basically I've created a new module that adds Web Seeds support.

The modifications in the core were minimal until now:

• Metadata parser (updated)
• Torrent object model (updated)
• Extracted some interfaces to allow specialization/extension

I've also created some implementations/extensions of some objects of core to accommodate the behavior of WebSeeds. And override the bind of some factory used to provide those objects. Just to name a few:

• WebSeedPeer (Specialization of Peer)
• WebSeedPeerConnection ( Specialization of PeerConnection)
• WebSeedsPieceSelector (Holds the piece selection appropriated to WebSeeds)
• WebSeedsPeerConnectionPool (Based on peer 'type' what kind of connection should be provided)

Some of theses implementations still need to be polished.

At the moment I'm working around the Assignments, to take advantage of WeebsSeeds connections to request bigger blocks of data instead of requiring one by one.

[atomashpolskiy]

In fact it doesn't look like you need hints :D Sounds crazy good, I'm really excited to see what you have in there

[atomashpolskiy]

@jorgemjorge , hope you are well! Have you been able to make any progress on this? I'd love to try and help, if you got stuck

[jorgemjorge]

Hi, I've been been on busy the last two week and didn't get around to make any progress. Hopefully I will be working on this next week. Thanks for your support, soon as I have something ready I will post my progress

[disconsented]

Have there been any news on this? Judging by the time of the last comment I'd say its safe to assume that @jorgemjorge has lost interest in it.