Support for operational transformation sync instead of full-page sync (page merging)

[jcgurango]

I use SilverBullet regularly from multiple devices - phone, multiple laptops, and my main PC. For a use case like this, the syncing feels a bit lacking - because changes need to go from device -> SilverBullet -> other devices, there's a bit of a time delay which usually isn't very long but sometimes it's long enough to disrupt anything I was writing. Additionally, if a device is temporarily offline for a few moments I could be typing something out only to see it disappear when it comes back online. I think many of these issues can be solved by implementing some sort of page merging, similar to how Google Docs or live TipTap editors do it. CodeMirror already has support for something called Operational Transformation so I think it could be integrated into SilverBullet. The basic principle is something like this:

1. There is a central authority, in this case the SilverBullet server.
2. When a client first opens a page, it will retrieve the full page contents.
3. From then on, all edit operations will be recorded as atomic updates.
4. The client will also listen and wait for any updates from the server.
5. On sync time, instead of sending the full page contents, the client will send its pending updates.
6. The server will then apply these changes to its own version of the page, and forward the pending updates to other clients.

For updates made to the server file system directly, the server would technically also be the client here (it would do the atomic updates and then forward that onto the clients)

I could implement this myself, but I'm opening this issue to get everybody's thoughts, any drawbacks you all might foresee, and any history of trying to implement this so I can avoid any mistakes that may have already been made in the past.

[zefhemel]

I had an implementation of this in the past (there's various CM plugins to implement this) but ultimately I removed because it added too much complexity to the system. There were many edge cases, sometimes memory leaks. These systems don't tend to be built for being offline for longer periods (even days and multiple sessions) so you'd have to add that. At that time I deemed it more trouble than it was worth and rolled back to the simpler system we have today. It has its issue (as you describe) but it keeps the whole system manageable, complexity wise.

At some point I may get back to this, but for now I have other priorities.

[jcgurango]

Gotcha. Would you be open to me implementing something custom that could work for specifically for silverbullet? My idea is to use a package like textdiff-create to create patches and have the server work off of those. We can keep the current way of pushing changes to the server. I'm thinking of something similar to doing git merges:

1. In the PUT request, the client will send diffs computed against the last time the page was updated from the server (let's call it lastRemotePageContent). Here we need some kind of tracking for what the last client version is, too. The simplest way would be to just send the value of lastRemotePageContent to the server and maybe put the diffs in a header or something, but we could also track it on the server side. Let's call this lastClientPageContent.
2. If the PUT request is accepted (OK response), the client will consider all those changes as "committed" and replace lastRemotePageContent with the current content of the page.
3. The server will compute an additional diff: the current content of the page (as it is on disk) against lastClientPageContent. Then we can just use textdiff-patch against lastClientPageContent + server changes + the diff that was sent by the client. This should finalize the page content with everybody's diffs intact.
4. The next time the client retrieves the page, it'll compute a diff of that against lastRemotePageContent, and a diff of the currently loaded page content against lastRemotePageContent.
5. The client will then replace the current page contents using lastRemotePageContent + (server version vs. lastRemotePageContent diffs) + (local version vs. lastRemotePageContent diffs) and replace lastRemotePageContent with the contents retrieved from the server.

This should make things pretty stateless and not require tracking large complex changes as single atomic operations.

[jcgurango]

So after some more thinking and playing around with some different ideas, I realized my main gripe with how this works is not really about the way things sync, but more about what happens when the client receives updates from the server - i.e. it wipes anything I was writing up. So, I've raised PR #1009 which tries to address this by still doing a merge - but instead of any server involvement it's purely a client-side merge.

[zefhemel]

Never say never, but I have no plans to switch to transformation transforms.