wtp-ext
wtp-ext copied to clipboard
tom-james-watson
Support for mutable torrents, bikeshedding the protocol name
First off, this is amazing. I've been wanting an extension like this for a few months now and am glad you beat me to it. 😍
I've been looking into how Torrents could be used for a p2p web, and found that an important question to answer would be how to update them in a fully p2p fashion.
With that in mind, I found BEP 46 and made a library which adds support for it to WebTorrent.
I think it'd be really cool if you included mutable torrents in your roadmap since it would get rid of the need for things like DNS TXT records which are still inherently centralized and require people to purchase domain names.
I've also been playing around with loading content out of torrents in a web context with my webrun project. Basically, I created a JS environment that has a bunch of browser APIs but is ESM-first and can be loaded the way Node does. I initially added support for loading JS files from IPFS and Dat, but I've started work on adding support for Torrents.
In order to load torrents, I needed to have a protocol. I was planning on sketching something up in Webrun and then submitting a new BEP to make it part of the standard. I settled on using the btih:// protocol for loading content from regular torrents after the urn:btih:<infohash> format used in magnet links, and btpk:// for mutable torrents for the urn:btih:<publickey> format used for mutable torrents.
I was wondering what you think about using these names and whether you'd be interested in contributions to the project.
Hello - great to see someone else excited by the idea of this!
I would definitely like to add support for domain name resolution using DNS TXT records regardless, as it is still nice to be able to have readable addresses and there's not really any other way to manage that. I don't see a proposal for mutable torrents clashing in any way with implementing that, however.
I wasn't aware of the idea of mutable torrents, so that's super interesting. What you're suggesting is especially interesting as it would allow for loading of torrents both by public key and also for standard infohashes.
Yeah, I would 100% be interested in contributions to this project. I would like this to be a community effort and am under no illusion that I would be able to build this well by myself anyway!
https://github.com/RangerMauve/mutable-webtorrent looks like a great start. I guess what we'd need would be:
- cli tool for creating / updating a mutable torrent, i.e. and extended version of https://github.com/webtorrent/webtorrent-hybrid
- A drop-in replacement for webtorrent on the client side. Looks like https://github.com/RangerMauve/mutable-webtorrent would potentially already be able to do that?
I am in no way married to the current wtp:// protocol, so if we were to go in this direction then I think your btih:///btpk:// proposal makes pretty good sense. I'm not sure how domain names would fit into that ... maybe btns://.
100% agreed with the DNS thing.
I was thinking that once I had proved that mutable-webtorrent works, I would do a PR to webtorrent proper and get support out of the box. mutable-webtorrent has the same API as webtorrent, so you should be able to use it client-side. The biggest limitation is that mutable-webtorrent requires the DHT to work properly so it doesn't work in the browser. Does wtp-ext use the net/etc implementations of libdweb, or is it acting like a browser version of webtorrent? Ideally it should be running a hybrid client.
Regarding btns / btih / btpk, maybe having multiple protocols is a bad idea? I really like how wtp:// would easily detect infohashes / DNS without having to have separate protocols. Maybe there's some way of signaling that the hostname is a public key instead of an infohash?
An infohash is always going to be 40 characters long, and public keys are likely going to start at 64 characters. Then you could have wtp:// (or maybe bt:// since it's BitTorrent) for all types of links.
Ah right, I forgot that DHT is node-only. Friday problems. That's a shame. So yeah, wtp-ext is simply acting like a browser version - it is using a default list of trackers.
I need to do some more reading on how exactly DHT works in order to work out what would be needed to get it running in-browser. From a cursory look, this might be of interest - https://github.com/libdweb/dgram-adapter. I'm not sure what other node-only dependencies webtorrent has for DHT other than dgram.
The exact protocols we can work out later, I guess. I was hesitant to just call it bt:// as it's not really just plain bittorrent, but an specifically opinionated way of resolving requests from bittorrents.
I think that you only need to support net and dgram since those are the parts that touch the network. I think @sammacbeth was looking into it and found that adding support for those libraries wouldn't be too hard.
I think @gozala would know more about it, too.
Yeah, I get what you mean with it being more than just bittorrent. I don't think it should necessarily be tied to webtorrent though since I could see other BitTorrent implementations supporting the protocol once this is this stuff is turned into official BitTorrent Enhancement Proposals.
@tom-james-watson I have started dgram-adapter with a hope that someone from community will pick it up and actually drive it. I have not had a chance to look at the work that @sammacbeth did but I would suspect he's work is more complete as he got dat working on top of it.
Here is the pointer to the work he shared: https://github.com/sammacbeth/browserify/commit/7046af105ad403610d1859a738b9ae9a9f4f95a5
Looking at the code, it's pretty solid. You don't even need to use his browserify fork, you just need to copy the implementation files to your project and add a browser field to your package.json that maps dgram, net, and dns to the appropriate file paths. Though you'd also need to explicitly require the node entry point instead of the browser one since browserify will automatically resolve webtorrent to the browser entry point.
@RangerMauve I as I as could tell the browser field only works for dependencies directly used in your project, and not for imports inside node_modules. Thus if you are using a module which internally uses net and dgram, this method won't work. This is why I had to create a fork in the end - I couldn't find a way to just provide the implementations that would be used throughout.
I should get a chance to look into this properly towards the end of this week. Thanks for the input so far! @sammacbeth - seems like a strange limitation - have you tried symlinking the dependencies so that they're not considered part of node_modules?
@tom-james-watson It's due to the way Browserify works. It looks a the browser field in the package.json in each module and has that override whatever the current aliases are. I don't think symlinking would help too much since browserify is explicitly being set to ignore certain modules.
OK so I've made quite a lot progress today!
I appear to have DHT working in the browser. See https://github.com/tom-james-watson/wtp-ext/commit/7bda62f7e00063f537d6420ecd23900db1b58966
In the end it was fairly easy to globally alias net, dgram and dns using webpack - https://github.com/tom-james-watson/wtp-ext/commit/7bda62f7e00063f537d6420ecd23900db1b58966#diff-11e9f7f953edc64ba14b0cc350ae7b9dR14
I had to fork webtorrent and torrent-discovery to stop them from excluding node-only modules when in the browser:
https://github.com/tom-james-watson/webtorrent/commit/21036dcc7fcb5839bdce18d6accf7b5cd0794724
https://github.com/tom-james-watson/torrent-discovery/commit/d3902386c6c7a60957d300b9a23469934270e261
I also had to make some slight changes to dns and net to get them working.
My fork of webtorrent, however, appears to have somehow broken wtp-ext. I've not had time to properly debug what's going on, but the data is not properly streaming back into the requests and is somehow all ending up in the tab's titlebar (WTF?):
I'll have to come back to this to find out exactly what is going on, but shouldn't be too hard to fix. I imagine webtorrent is doing something differently as it now thinks it's running in node.
Anyway - that screenshot there is a torrent seeded by the standard DHT-only webtorrent loading "successfully".
You can check out the work on the node-polyfills branch, should anyone feel like debugging what's going on with the request data streaming!
Hah ok, definitely time for me to take a break. Turns out I just had a typo in my test HTML ...
Exciting!
Next step would be to try and clean up this branch's code and to work out what to do about the changes necessary to webtorrent and torrent-discovery.
That's so cool!!
It'd be super easy to port over the stuff I did in mutable-webtorrent into this, and then you'll have updatable websites loaded from BitTorrent in the browser with, I think, minimal code. 😍
Nice, I didn't know you could do this with webpack, seems simpler than browserify. Does it also polyfill other node APIs like Buffer and path?
Buffer is actually implemented when running an extension in Firefox - I've not done anything special there. As far as I can tell Buffer is simply an extension of Uint8Array anyway, which is already available in the browser. I assume that path is simply not used with how we are using webtorrent - I'm not polyfilling it.
Downloads when connecting via DHT are massively slower and the connection between peers seems to be very intermittent. Sometimes it won't work at all or downloads will stop part way through.
Have not been able to debug what is going on exactly.
I've picked up development on this again by the way. Here's what's changed in the last few weeks:
- Added support for domain resolution, e.g. can now load
wtp://tomjwatson.com. - Added a proper browser action so you can see the status of the current torrent
- Added a torrent management page where you can see what torrents you're seeding, delete them etc.
And here's what's in the works at the moment:
- Switching to use https://github.com/random-access-storage/random-access-web for persistent torrent storage
- Turning off seeding by default. Instead, torrents will be removed from storage after all tabs are closed and a TTL expires. You can choose to seed a site, in which case it will be seeded forever.
I've also updated the node-polyfills branch to use latest versions of libdweb. This branch has working DHT discovery and TCP data transfer, so is compatible with traditional torrent clients, which is pretty huge. I've narrowed down the connection issues to the peers choking the client (or viceversa, I'm not sure).
If anybody has any experience working with webtorrent or torrents in general and wants to help debug what's going on, that would be amazing. We're so close to having a dweb that is 100% compatible with the whole torrent ecosystem, which is so exciting.