Feature request: offline-first (or tutorial)

[bionicles]

Dear Team Userbase,

Thank you for making this platform!

One thing I hacked, and got stuck in the weeds with, was offline-first. I did manage to hack it, but it's not exactly an ideal solution -- we store the app state in a single entry in userbase, then use ramda.mergeDeepRight to merge the one with the later timestamp into the one with the older timestamp. It feels like a hack, and this could lose data, because it's Last Write Wins instead of some other option. Also, I doubt it's highly performant or scalable to stuff the whole user data blob into a single key.

Would it be possible for userbase to add some offline-first function, or perhaps a tutorial to figure that out? It would be incredibly cool to have a sort of CRDT / ORDT so userbase apps would 'just work' offline, and sync automatically. I dabbled with automerge but it didn't work with ramda.

What do you think?

Thanks in advance for continued development of userbase!

Bion

[j-berman]

Native offline-first functionality is definitely something we're thinking about for the future! That being said, the underlying Userbase data structure is quite powerful, and what you're trying to do with Automerge should be possible to do in a decently performant way!

A deeper explanation of how Userbase works under the hood, and why I believe it can be relied on to work with operation-based CRDTs: when a client calls any of the database write operations (insertItem, updateItem, etc.), the client pushes the operation up to the server, the server then assigns a sequence number to the transaction based on the order it receives it, and then broadcasts the transaction out to all clients. Clients therefore receive transactions in a guaranteed, durable order, and never receive duplicate transactions. Which I believe is a requirement for operation-based CRDTs (edit: the no duplicates part). This process is described further here.

What the above all means is that you can rely on insertItem exclusively to insert snippets of Automerge changes into a database, in a way that all clients will receive the changes without any duplicates. So, you can do something like this:

Writing

let localCollection = Automerge.change(syncedCollection, doc => {
  // make arbitrary changes to the collection locally
})
let changes = Automerge.getChanges(syncedCollection, localCollection)

// now push changes up to Userbase
Promise.all(changes.map((change) => userbase.insertItem({ databaseName, item: change })))

Reading

let syncedCollection = Automerge.init()
const databaseName = 'conflict-resolving-db'
const changeHandler = (changes) => {
  // syncedCollection will automatically remain in sync with the server relying on Automerge's conflict resolution
  syncedCollection = Automerge.applyChanges(syncedCollection, changes.map(change => change.item))
}
await userbase.openDatabase({ databaseName, changeHandler })

Note that all the changes would need to fit in to the client's memory for this to work, but I believe if you're working with Automerge, you make that assumption anyway since each item holds all historical state.

All the building blocks are there!

[j-berman]

I made an offline-first Google Docs alternative to show how to do this! It's a bit complex - planning to provide a simple tutorial on how to create a super basic version of that^ here. In the meantime, here are the relevant areas of the code:

• Store Automerge changes in IndexedDB using Dexie.js
• Push those changes into a queue, to be stored server-side using Userbase
• Push the changes to Userbase from the queue every 1 second interval
• When receiving Automerge changes from Userbase that a client has not seen before, it'll store them in Dexie.js

[bionicles]

@j-berman epic! hey if you want somebody to proofread your tutorial i'm happy to help out - bion at bitpharma dot com