datproject-discussions

datproject-discussions copied to clipboard

It seems any Dat plugins so far require manually running a process in the background. Is it plausible to have a major browser plugin for Dat that doesn't require that?

I think the easiest way forward for that is to use a public gateway kinda like http://gateway.mauve.moe:3000 and have that bundled in the extensions by default.

I'm planning on supporting plugin-less Dat interaction by relying on the gateway to inject the DatArchive API into the HTML of a page.

The biggest limitation is that discovery-swarm won't work in a browser plugin due to the environment differences (no TCP/UDP/MDNS).

It seems any Dat plugins so far require manually running a process in the background. Is it plausible to have a major browser plugin for Dat that doesn't require that?

Mhh I think you could do that by mapping fs access to some kind of browser-only storage (I think that @pfrazee did some experiments on storing blobs in indexedDB). As long as there is no reliable way to store data inside browsers it's hard but non impossible :).

I've choose to add a websocket-backed gateway that should give full dat-power in the browser. I'm successfully reading files managed by dat from the browser. Didn't published the full documentation yet but the code is available here if you want to take a peak!

Recently @sammacbeth built https://github.com/sammacbeth/dat-fox which has a dat protocol handler. This handler simply uses an http gateway to render dat pages in the browser. It's also a good place to look !

As mentioned, there are several limitations in the browser extension APIs and environment preventing this:

1. The protocol_handlers API only allows a protocol to be registered as a redirect to another url. As you also cannot register a function as a protocol handler, nor run a HTTP server inside an extension, this prevents the extension from serving the protocol itself. With the old Firefox Bootstrap extensions all of this was possible, however these can no longer be used with Firefox. The Firefox Addons team are generally receptive to new API proposals though.

2. No socket API. Extensions are limited to http(s), websockets and WebRTC, and so would not be able to communicate with nodes running node dat.

3. No file API, though IndexedDB could provide a file-like abstraction. This storage is, however, potentially volatile, as the browser may impose storage constraints, and it is very easy to accidentally delete.

Given these limitations, for the near future a secondary process is required. On the dat-fox I am proposing limiting the technical overhead of this, by installing a Native manifest for the gateway executable in the browser, such that, once installed, the user does not have to launch this process themselves. The browser automatically launches the process when the extension is started (this is part of the Native messaging API and also available in Chrome).

Public gateways are another option, but they introduce a significant privacy issue as the operator can observe all of the dat browsing of users. There are also many cases where it is required that dat urls should be kept secret to prevent unauthorised access to the content, and a public gateway would not be safe for these use-cases.

Nice input! Tyvm about the Native messaging API thing, wasn't aware of that possibility!

Public gateways are another option

Like https://hashbase.io/ I guess. Although they kinda break the p2p utopia in which they are centralized?

I see public gateways as a temporary solution to just bring the technology to people with as little effort as possible.

If you can make something using dat and share it to people that might not know about it yet, you could create network effects and get people more interested in installing extensions to get better support (performance, privacy).

Getting people to check out new web apps is hard enough, it's even harder if they need to install and configure some sort of software first. A gateway can get your Dat-based applications in front of people now and spread general usage of these apps and give incentives for browsers to support the APIs and protocols.

If y'all haven't seen it, some good work going on here: https://github.com/goonism/hyperproxy/ to support out of the box Dat browsing via a gateway + webrtc

https://hyperproxy.network/

I like the idea behind hyperproxy, but IPFS has tried to use WebRTC for P2P stuff and the performance was crappy which made me kinda skeptical of using WebRTC for anything large scale (say more than 10 peers).

Signaling servers are a point of centralization which discovery-swarm eliminates by using the DHT and MDNS, and you can't have WebRTC without them.

If all of dat switched to WebRTC and WS for transports, and non-browser clients had signaling servers built into them, then this approach would be a lot more scalable. Otherwise I think that getting first-class browser support though the existing dat network is going to be more important.

Of course using WebRTC and signaling servers is a hell of a lot more decentralized than dat-gateway. :P

I'm gonna be working on enabling the Bunsen browser folks to have a more complete solution and working on features to make a PWA that uses dat-gateway to allow people to use dat:// urls without installing anything and using a web app instead (to enable casual users to be exposed to the concept). I'll have the web app try to persuade people to install any extensions or browsers that would be better suited for their platform.

I'm helping out with the Bunsen efforts and believe all these directions are worth exploring. Currently Bunsen UI looks for a local dat gateway first localhost:3000, if it doesn't find one, then it falls back to @RangerMauve's gateway. We also published the Bunsen UI as a Dat so the Android App which is bundled with a local gateway can serve the UI through the gateway. That means in any browser we can also see the Bunsen UI by hitting an online gateway -> http://gateway.mauve.moe:3000/bunsen.hashbase.io/

That puts us two steps away from having a Bunsen PWA. To get browsers to prompt users to install PWA we'll need to add the necessary manifest file to Bunsen UI and also get the gateway on https. We're close and very excited about the potential of getting DatArchive API working!

I've added my initial implementation of redirecting Dats to use subdomains

https://github.com/RangerMauve/dat-gateway/tree/subdomain-redirect

Run it using node bin.js -r and navigate to http://lvh.me:3000/40a7f6b6147ae695bcbcff432f684c7bb5291ea339c28c1755896cdeb80bd2f9 to get redirected to http://82kzddgmfbk9bf5wzx1jyu2cfeujj7n37718r5unh5pdxe0bubwg.lvh.me:3000/

@rjsteinert Have you looked into using the registerProtocolHandler API to register a webpage as a protocol handler? Maybe that paired with an iframe loading gateway content could be enough to make a web app that loads dat:// URLs through the gateway.

Maybe that paired with an iframe loading gateway content could be enough to make a web app that loads dat:// URLs through the gateway.

I had a successful experiment with this.

@soyuka That's awesome! Do you think you'd be interested in setting something up that works with my public gateway http://gateway.mauve.moe:3000/

I've updated my public gateway to support the redirect to subdomains, but I've got a bug that means DatDNS doesn't work any more.

Disregard the DatDNS thing, got it working. :D

I think there should be a different default port for the gateway since 3000 is pretty generic and can clash with other services. I suggest port 3495 which is 0xDA7 in hex, which kinda looks like DAT.

@RangerMauve Looks like decent support for navigator.registerProtocolHandler. That's very similar to how we handle dat:// links in Android using Intents API. I wonder if Chrome will allow handling of dat protocol, probably not right? Firefox though :). GIving it a try.

Awesome job on the gateway! Not only does that increase security immensely, it fixed a bunch of sites that wanted to live at the root of a domain.

@rjsteinert I also updated the gateway and dat-archive-web To support DatArchive.resolveName and improved detection of HTTPS. Check out this branch.

Version doesn't work at the moment due to url parsing shenanigans.

Next gonna work on getting fork to work.

DNS URLs in the constructor don't work yet, so I might address that before fork.

Not sure what to do about commit/diff/etc.

Eventually I'll need to set up an SSL certificate. Might be a pain since I need a wildcard.

As expected, Chrome doesn't play well with apps registering as protocol handlers for dat://.

Firefox I bet works, but now I'm realizing our Web Components based Bunsen UI is having trouble in Firefox. Time to work on that :).

For SSL, I've had luck with some Lets Encrypt tooling that makes setting up a proxy with SSL support with Docker pretty easy. Recently I remember reading they added support for wildcards. An example -> https://github.com/rjsteinert/serve-a-static-site-with-ssl-example

In the mean time use web+dat or ext+dat!

Interesting topic also: https://news.ycombinator.com/item?id=16242336

Thanks @soyuka for the interesting read. We got Bunsen working for Firefox and see that registering to protocol handler is working.

You can try it out on the gateway I just started http://104.236.57.117:3000/bunsen.hashbase.io/

Bunsen it not compatible with @RangerMauve's recent update to his gateway but we're looking forward to being compatible ASAP!

@rjsteinert oh no, did I break something?

Probably me! I've been working on Bunsen in the past hour trying to figure out a URL issue I'm starting to think is related to version of dat-gateway we're running for Bunsen. Btw, I love this "bug" I found.

Shall we continue our discussion on dat-gateway in your repo's issue queue? (The issue queue is currently disabled)

Oh snap I thought I fixed that. Enabled issues. I'm about to fall asleep though so you may need to wait 11 hours for me to get back on it. :P

LOL yes it's fixed! That was gif from earlier today. Time flies! Have a good rest.

I've released version 1.0.0 of dat-archive-web to NPM.

Everything is supported now except for archive.diff(), archive.commit(), and archive.revert(). DatArchive.selectArchive doesn't work for the default version, but it's easy to add behavior to support it.

Next step is going to make a dat-polyfill which will do link renaming (for images/iframes/anchors linking to dat://) and provide the DatArchive global which will attempt to either connect to the local gateway or the public one, and will coordinate with an iframe to do a rudamentary selectArchive dealio.

@rjsteinert Do you have time to talk about this polyfill stuff? I think this is basically what bunsen-ui is trying to do. I'm going to do the initial work for the polyfill, but I'll want to talk about persisting data between pages and how selectArchive should work.

Bunsen handles URLs in three scenarios...

1. You are in the Android Bunsen App, using Intents API the app is registered on the system level as a handler of dat:// links.
2. You are in a browser that supports navigator.registerProtocolHandler https://github.com/bunsenbrowser/bunsen-ui/blob/master/src/bunsen-app/bunsen-app.html#L140
3. You are not in any of those so start the anchor link rewrite party! https://github.com/bunsenbrowser/bunsen-ui/blob/master/src/bunsen-app/bunsen-app.html#L145

How about 9pm ET tonight at https://talky.io/dat-polyfills ? All are welcome to join.

21:00 ET works for me. :D

Also, regarding option 3, I suggest using a MutationObserver which filters for nodes with the src and href attributes paired with an initial change instead of polling twice a second. I think that would be more efficient and eat less CPU