Feature request: streaming torrents from disk and from memory

[cocool97]

I've got two questions about rqbit and its public API. I'm not interested in using the executable but would like to use the library directly (to use in another personal / open source project).

1. How can I download a torrent to something other than a directory using it ? Could it be possible to implement downloading into a Writer, that could be a directory by default ?

2. I do not think this is possible yet but would it be possible to implement a Read / Seek interface on ManagedTorrent ? This would allow for streaming / downloading only parts of torrent files (like done in https://github.com/anacrolix/torrent, but for Go @anacrolix) ?

Thanks for you time and the amazing work !

[ikatson]

Hi, both aren't possible at the moment.

1. Could it be possible to implement downloading into a Writer, that could be a directory by default ?

I'm not sure it makes sense without the second feature you're asking for. Torrents are multi-file (potentially), so one writer won't work, so what does it mean to write to smth else, when I'm requesting to download a torrent with 50 files?

Conceptually, the operations that are working on the FS today are:

• join filename with the base path to download to
• open a file in either read-write, or read-only mode
• seek
• write
• read

So it shouldn't be hard to write an abstraction where you provide all these somehow. But again, I'm not sure what's the use without streaming. And streaming probably makes sense only for one file at a time.

2. would it be possible to implement a Read / Seek interface on ManagedTorrent

Some building blocks are there, but you'll need to rewrite a big chunk of TorrentStateLive (which is big). The whole ManagedTorrent thing - which is a state machine "Initializing -> Paused -> Live", doesn't make sense for streaming.

So it seems like it should be a completely different API.

But if you combine the "make_peer_rx" with a custom implementation of PeerConnectionHandler + something that spawns peers, and queues needed pieces to them (TorrentStateLive does this today), then it could cover it.

So in short, no, it's not available today, and is a pretty substantial amount of work, but possible.

[ikatson]

Btw @anacrolix, as you were contributing recently, maybe you have some thoughts on it, especially as you implemented another streaming client already

[anacrolix]

I can only add that yes, it would certainly be possible, and it would end up looking much like how I did it in https://github.com/anacrolix/torrent. It's not trivial as it infects how request prioritisation, torrent state, and storage are handled.

What you should definitely not do is download to files and then try to guess the state of the file and stream from disk without synchronization.

As a workaround you could do the streaming part using a HTTP wrapper around https://github.com/anacrolix/torrent, like https://github.com/anacrolix/confluence. That would let you use Rust but do the streaming parts in Go (if that's palatable to you 😆, it's not particularly tasteful to myself anymore).

I have considered moving to Rust with my streaming client/knowledge but I don't have adequate funding for the effort. I think rqbit could support it, the performance and fundamentals are very good.

[ikatson]

@cocool97 I've done both in https://github.com/ikatson/rqbit/pull/128, still testing, but seems to work fine

Look at examples/storage.rs to for an example on how to pass in a custom storage, and in test_e2e_stream on how to stream a file.

Streams are also available over HTTP /torrents/<id>/stream/<file_id>.

Curious if this helps. I'll polish and test a bit then merge them into main.

Don't know about storage backends, but streams are cool as you can paste the URL into e.g. VLC and jump around a video file way before it's fully downloaded.

[cocool97]

I'll continue the discussion here as you merged #128 and #129 !

Your API is fine, the notion of id and file_id seems OK. I've tested your code (more precisely the http_api) and this work fine when the file is available on the local fs. If we want to avoid having storage on the fs, what would it take to ask peers for new data when the stream is seeked ?

[ikatson]

I've tested your code (more precisely the http_api) and this work fine when the file is available on the local fs. If we want to avoid having storage on the fs, what would it take to ask peers for new data when the stream is seeked ?

It already will request new data from peers when the stream is seeked (as long as the torrent is unpaused). And if you want to avoid having storage on the fs, just implement a new storage that suits your needs

[cocool97]

I've kept testing your code with my torrent streaming project. Here is the URL is you want to take a look at it (URL) Integration was pretty easy and straightforward. I took your code to handle streaming server-side and your InMemoryStorage. I just found an issue which is that all torrent data is stored in memory... This is an issue because torrent data can be pretty huge.

Technically, I think that is comes from the fact that torrent reading from peers is not driven by client-side (VLC e.g), meaning that both sides are decorrelate and this causes this issue.. Cannot storage be used only for readahead and let the streaming be done from clients directly ?

[ikatson]

The inmemory one is an example. You can write a storage that will work in a different way, or just use the disk one. Here's some other example storages I was toying around with in another branhc https://github.com/ikatson/rqbit/tree/piece-cache/crates/librqbit/src/storage/middleware

However if what you're asking is if you can get away without any storage altogether and just stream from network, then the simple answer is it's not possible yet. Packets come in random order, and you need some kind of reasssembling buffer at least. I tried creating some "garbage collecting" storage, which will remove pieces that are no longer needed by streams. It kind of worked, but was bugged, so I abandoned the approach eventually to redo someday.

[ikatson]

Also from the bittorent protocol perspective it's not great to just download and not reshare. That's why other clients don't have that feature either.

So what I suggest to you is just use the default disk storage, unselect the files you don't need (by passing "only_files" explicitly), and then stream. It will download the file in background, and if streams are alive, their pieces will be prioritized.

[ikatson]

Closing as this is done in 6.* versions

[izissise]

Wow that's awesome Do you have the link to the documentation?

[ikatson]

Wow that's awesome Do you have the link to the documentation?

@izissise here's an example of how to stream a file in librqbit https://github.com/ikatson/rqbit/blob/main/crates/librqbit/src/tests/e2e_stream.rs

HTTP API: /torrents/<id>/stream/<file_id>

Web UI: once you click on the "cog" icon, the video files will point to the above HTTP API link. You can paste them into e.g. VLC or ffmpeg, and then you can seek in the video file while its being downloaded.