Using no disk based files with torrent creation.

[BernhardGlueck]

Hi !

I have the following issue, or rather just a request for someone to point me into the right direction. I have a content repository that is read only from my side, and which is represented by the following interfaces. Each content file is represented by a sequential collection of blocks all identified by their SHA256 hash. The blocks are each 1 MB in size ( except the last one of a given content item )

interface IBlock 
{
  long Size { get; }
  byte[] Sha256 { get; }
  
  Stream OpenRead();
}

interface IContent : IReadOnlyList<IBlock>
{
    string Filename { get; }
     long Size { get; }
}

Now i would like to distribute those content files via bittorrent, using monotorrent of course. Backward compatiblity is no concern, as all clients are under my control. So the V2 protocol can be used,

In my mind the structure i have maps perfectly to torrents and their piece structure, and since i already have the SHA256 hashes this would also work quite well with the V2 protocol .

However currently i am struggling on how to plug my system into montorrent e.g creating a torrent file from a collection of such content items...

All the samples and interfaces like ITorrentFileSource etc assume that the files reside on disk, and that piece are not already defined.

Is there a way to plug my system into the torrent creation pipeline ? Where should i start to look in terms of interfaces ?

[BernhardGlueck]

After some more investigation it seems that i would have to reimplement TorrentCreator However a lot of the types needed to do this are internal to monotorrent... i would rather like to avoid doing a full source fork ... Any ideas ?

[lostmsu]

@BernhardGlueck what did you end up doing?

[BernhardGlueck]

I wrote everything from scratch so its now not related to monotorrent anymore ( our own Beencode writer and of course the Torrent V2 creation ) .. works like a charm... however it was very difficult as the TorrentV2 spec is very ambiguous and left us with a lot of trial and error to do. Especially if you don't have the original file but only its piece hashes ( tree padding logic and the 16 kb page hashes )

[alanmcgovern]

Ah, it's actually quite easy to do with MonoTorrent. It's unlikely you'll want to return to MonoTorrent, but if you are interested the extension point you need is also used in these (new) tests:

The first part is to create a custom IPieceWriter. You need to implement the 'Read' and 'Write' methods to read/write from your content repository. https://github.com/alanmcgovern/monotorrent/blob/master/src/Tests/Tests.MonoTorrent.IntegrationTests/IntegrationTests_FakePieceWriter.cs#L29-L109

The second part is to use that writer when creating torrents: https://github.com/alanmcgovern/monotorrent/blob/master/src/Tests/Tests.MonoTorrent.IntegrationTests/IntegrationTests_FakePieceWriter.cs#L214-L222

Hope that helps the next person at least! Good luck with your project too!

[BernhardGlueck]

Thank you for your answer.

We are working on our client side right now, which downloads the torrent files we create. Right now we are using a libtorrent custom wrapper but that quite complex and error prone, and also requires us to ship native dependencies for all supported platforms ( 6 )

So switching back to monotorrent for the client side would be great, however we can only support V2 torrents ( due to SHA256 requirements for our hashes in our backend )

I was under the impression that V2 Torrents are not yet supported in monotorrent for downloading ? If that has changed it would be an real option

[alanmcgovern]

I was under the impression that V2 Torrents are not yet supported in monotorrent for downloading ? If that has changed it would be an real option

Do check out the latest preview release in nuget, or build the latest master commit. While it's probably not fully done-done yet, I have just shipped a bunch of fixes for various aspects of V2 support. I'm fairly convinced that, at the least, MonoTorrent is compatible with whatever version of libtorrent 2.x that qBittorrent uses in terms of being able to distribute the merkle tree, and also download/upload data for v2 only torrents.

We are working on our client side right now, which downloads the torrent files we create. Right now we are using a libtorrent custom wrapper but that quite complex and error prone, and also requires us to ship native dependencies for all supported platforms

Native dependencies can be tricky alright. It's been a while since I had to worry about p/invokes, but it's something I definitely do not miss ;) If MonoTorrent suits your needs then great! If not, do let me know what the issues are and I can figure out what it'd take to address them!