WIP: Sync

[pgte]

Allow sync between nodes.

• [x] save log entries to IPFS
• [x] broadcast head
• [x] listen to remote head broadcasts
• [x] merge remote logs

Addresses #1

[Bajix]

What's blocking this from moving forward / what is there still to do? I could see this project pairing really nicely with PouchDB.

[mikeal]

I could see this project pairing really nicely with PouchDB.

I don't quite see how that would work. The data models are very very different. I've written adapters from lots of datastores to sync to couch/pouch but I'm struggling to figure out how I would build one for IPFS.

[Bajix]

I presumed it was the case that PouchDB would work with anything that implemented the abstract-leveldown interface, and thus were this to entirely do so things would just magically work. It’s not clear what additionally if anything would need to be done after the interface is implemented.

IMO IPFS backed DBs could be incredibly useful for making offline-capable apps that sync user data between web/desktop/mobile/backup. Much more attractive alternative to the CouchDB one user:one db concept.

I think there is also work being done to do a PouchDB OrbitDB adapter, but I think that is also super WIP status.

[mikeal]

PouchDB doesn't do syncing at the level-down layer though. PouchDB is using level as the storage abstraction, it maps a syncable data model on top of that.

If you try to do syncing at the level layer with this you're asking for conflicts if both peers are accepting writes. This would work fine for read-only replicas but BadThings™ will happen if you're writing to two databases on top of this that have their own syncing semantics.

Much more attractive alternative to the CouchDB one user:one db concept.

Totally agree, that's one of the reasons I'm working in the IPFS/IPLD space instead of the Couch space nowadays :)

But, the CouchDB repo model is critical to how it does replication. There's work happening in CouchDB 3.0 to create "view based sync points" but it's still problematic because you have to know the data profile you want to sync for each user/device beforehand.

The nice thing about the IPLD data model is that you have a merkle graph and can pull whatever you need out of it dynamically and re-sync any parts of that graph in the future against another root node. If you start trying to map the Document/Repo model on top of this, and then start syncing it around with PouchDB, you'll lose that dynamic syncing profile.

In case you're worried I don't know what I'm talking about, I added CommonJS modules to CouchDB, I started the PouchDB project, and I've written a CouchDB compatible datastore directly on top of LevelUp ;)

[Bajix]

If you try to do syncing at the level layer with this you're asking for conflicts if both peers are accepting writes. This would work fine for read-only replicas but BadThings™ will happen if you're writing to two databases on top of this that have their own syncing semantics.

I can see how real-time syncing would be an issue. I was actually thinking about this to solve the use case of allowing a user to store arbitrary data, and then to sync up from another device later, such that there's only ever one device writing at a time. Basically a load+save model w/out proper real-time syncing capabilities that's really only suitable for single-user use.

The nice thing about the IPLD data model is that you have a merkle graph and can pull whatever you need out of it dynamically and re-sync any parts of that graph in the future against another root node. If you start trying to map the Document/Repo model on top of this, and then start syncing it around with PouchDB, you'll lose that dynamic syncing profile.

I'm aware; IPFS/IPLD are remarkable, and I'm definitely eager to build w/ that data model. You have the right idea in working in that space! If you have any resources/projects that build off of this data model that you can reference, that would be greatly appreciated. Thanks!