Merging with NextGraph

[joepio]

I had a couple of great conversations with @niko-ng who's working on NextGraph. We've been exploring some ways in which we could work together towards something greater by collaborating. Niko proposed using NextGraph in AtomicServer.

This would be very impactful, but also worth exploring.

This document will be updated / changed as I have more conversations with Niko

Why it would be cool (new features we gain)

• Local-first. That's a dream that I've had for a while. It's one of these characteristics that AtomicData doesn't have that would be very powerful, but is very hard to implement. NextGraph already has a lot of answers in this domain. #998
• Encrypted. NextGraph encrypts everything. #300
• Branching. NextGraph is a CRDT that allows multiple concurrent branches. This is almost there.
• Drafts. #129
• Real arrays. With insert / splice methods etc. Nice.
• Form submissions. In nextgraph, every document has an inbox where you can submit things. These things can be configured to have to conform to some shape. #875

NextGraph would replace...

• Atomic Commits as an abstraction layer (NextGraph supports multiple CRDTs). This also includes versioning, history, websockets, subscribing...
• Sled as KV store with RocksDB or some successor. We still need to investigate what this will be. But NextGraph will provide disk persistence.
• Our existing Query index mechanisms. OxiGraph would do the heavy lifting, searching with SPARQL. However, that doesn't have lexicographic sorting or full text search.
• Atomic Agents. NextGraph also has a crypto based user / agent model. Wants to use Opaque, retrieves a wallet (containing your keys) safely from a server. You can open that wallet in a bunch of different ways. NextGraph's Pazzle abstraction will soon be replaced with a username-password. NextGraph would provide the login screen.
• Invites
• How URLs / subjects are built. Resources in NextGraph have special properties which link dids to (human readable) URL paths.
• Schema validation (Atomic Schema). In NextGraph there is this thing called the "Verifier". It's a crate that runs verifications to make sure data is still valid. This should a new plugin, like an Atomic Schema validator plugin. We should consider doing JSON schema with NextGraph and extending that a bit. Will be a rust plugin.
• Authorization / persmissions. Atomic has read/write rights, as does NextGraph.

AtomicServer would still...

• Convert NextGraph resources to HTTP urls, similar to how ActivityPods are related to NextGraph right now
• Do pagination / indexing. The AtomicServer would be a Client which has its own agent, which has its own rights. The user can give access to subsets of their data.
• Do full-text search using Tantivy
• Provide the front-end + libraries.

Questions & Doubts

• How does Atomic Schema relate to what NextGraph is doing?
• Content Addressing?
• P2P? NextGraph.
• The performance will almost certainly take a big hit. There is no lexicographic sorting in Oxygraph at all, which means collections will be far slower. In addition to that, NextGraph is pretty slow when constructing a resource from commits. To what extent can we get good performance if we merge? What needs to change?
• NextGraph doesn't support querying over JSON yet, but it does support RDF indexing. Maybe we can index Atomic Data as RDF
• NextGraph doesn't have fulltext search yet. Niko considered using Sonic, I told him about Tantivy, which I'd recommend because of some more powerful features.
• Testing coverage of NextGraph is nowhere near that of AtomicServer. This doesn't mean the code is unreliable, but it does limit the extent to which I can mess around in it without feeling uncertain of it.
• Would it work with the Actix (tokio) runtime? NextGraph works with async-std, but that's no longer properly maintained. I'm not sure if one process can have both runtimes in parallel.

What would need to happen

• JS lib: use NextGraph JS lib under the hood. That deals with fetching and editing data, keeping things in sync.