Torrent Replica

[benbjohnson]

Per discussion with Keoni Gandall, it could be useful to distribute incremental updates for large public SQLite databases over torrent. This needs investigation to see whether this is actually a good idea or not. 😂

[tbillington]

I assume you're already aware of it, but just in case you aren't, syncthing might be some interesting prior art.

I looked into using it previously to distribute multi-GB VR applications to clients with spotty internet. Functionally the MVP worked, however it didn't come with much provisioning/remote management support and I was afraid I'd have to walk someone without technical experience through re-configuring syncthing over the phone :)

[benbjohnson]

I've seen syncthing before but I hadn't considered it for distributing Litestream data. That could be interesting although I don't see a lot of info on using it as a library. I suppose it may work when just running syncthing as a standalone binary. There's some upcoming work to change Litestream to use nearly all immutable files so that would probably help with syncthing.

The torrent replica is a little bit different in its use case though. Part of the idea with torrents is to offload bandwidth costs to other folks so it's not so expensive to distribute large, public databases. With synching, would you still be paying for all the bandwidth or would syncthing users share some of that cost by syncing between each other?

[tbillington]

I meant the protocol specifically, or the techniques the protocol uses. I agree bundling in syncthing whole wouldn't be ideal.

Syncthing does work like torrents in that you'll download and upload with all available peers, though I'm unsure on how it chooses who downloads/uploads how much, aka the 'fairness'.

Syncthing requires an agreement (clicking through UI or running a command, not just like a handskahe) from both clients before a connection is established though, it's not 'open by default' like torrents are, which is why I suggest the protocols and techniques as opposed to syncthing itself :)

[anacrolix]

I assume litestream publishes wal update files with some minimal metadata. You could create a mutable DHT item that points to the head of a list of immutable DHT items that contain a torrent infohash, and the DHT target for the previous update. Or you could fit the whole list in the mutable DHT item, going back 50 or so updates (without breaking out into a separate torrent for each update to the list in the DHT).

The mutable DHT item scheme allows for signing the updates, you would distribute the public key to subscribers and they would swarm around the torrents contain the WAL update files.

If you want to provide some pointers to some specifics on the updates I can determine the optimal solution for this.