jsonpatcher and undetermined db events ordering between logs

[requilence]

Consider scenario:

• 1st peer creates new threadId and collection
• then it Create new item in the collection
• 2nd peer joins the DB by addr
• then it Save the item with the same _id
• 3rd node joins DB by addr
• 3rd node executes Save before Create leading to Fatal error here

Here I have the example based on json-books: https://gist.github.com/requilence/a9d1c4ceab32e1f09b1ce2f3e895e242 It always fails with FATAL app app/app.go:185 can't save nonexistent instance

But the roots of this are much interesting. Do you consider any sorting of events between different logs? Here are some ideas:

• vector clock on each record(big overhead if you have a lot of logs, still need to solve concurrent scenarios somehow)
• parentId that will point to the last logId+recordID applied to this DB before this event

This way the client can reconstruct the partial ordering of events between logs and there will be much more space to solve conflicts in the Events Reducer

[jsign]

@requilence , thanks for your report!

Yes, there's a big limitation in the json patcher since it doesn't have any kind of causal ordering. Apart from the scenario you describe there're other "problems" such as the state of the db not converging depending on how saves are processed by different peers. Of course, all depends on how you plan to use the data and if that's a limitation.

We have plans to introduce CRDTs Event Codecs to give strong eventual consistency. So stay tuned!

[sanderpick]

Yep, it's a problem! There's two things that need to be added to really make this work well:

• Joining from a snapshot: https://github.com/textileio/go-threads/issues/263 (this would ensure that the instance is always present before any interactions)
• CRDT codec: https://github.com/textileio/go-threads/issues/51 (provide casual ordering)

[requilence]

@sanderpick @jsign Sounds cool! Just to make sure I got it right, the log records themselves will not be affected and the ordering mechanism will be on the level of Events Reducer, right?

Also, jsonpatcher has Timestamp encoded in the Event. It's far from the ideal solution but if we will sort events in the Reducer by Timestamp it will solve some of the surface problems like I have shown