Updates and deltas are posted before the data is persisted

[gsvarovsky]

LevelDB writes are "asynchronous" by default in writing to storage. So it's not enough to wait for the async LevelUp operation to complete before an update can be guaranteed complete.

quadstore does permit the sync option to be passed in the batch operation.

Ideally, the local transaction should complete as soon as LevelDB is happy (and release the dataset transaction lock), but the delta and update should not be sent until later. Unfortunately having both Futures is not available at any layer (quadstore, Levelup, LevelDB).

This should only be a problem if the clone OS dies - process death should be OK. So maybe better to make a clone tolerant of re-receiving its own updates during rev-up.

[gsvarovsky]

Failure mode: Clone has notified an update to the app, but fails to persist the transaction.

A UI app should not be persisting its own copy of updated state. Since this circumstance requires process death, the whole app should have died.

A database-sync app may or may not have finished pushing the change to the database. A message re-delivery during rev-up will be identifiable by a duplicate @ticks value.

• Spec: A database-sync app should not re-apply a transaction with duplicate @ticks.

[gsvarovsky]

Failure mode: Clone has notified a delta to the domain, but fails to persist the transaction.

• Test: Clone accepts one of its own deltas during rev-up (and correctly updates its clock).
• Test: Collaborator sends requester delta during rev-up.

[gsvarovsky]

Failure mode: Persisted local clock is out of sync with persisted data.

Should not happen if LevelDB batches are atomic as advertised.