equinox icon indicating copy to clipboard operation
equinox copied to clipboard

Cosmos: Provide ability to separate storage of snapshots

Open bartelink opened this issue 6 years ago • 1 comments

The unfolds mechanism, as documented in #50 provides the ability to efficiently stash relevant data facilitating reading of state based on a point read.

However, this is can be suboptimal in the cases of larger snapshot states:-

  • the data gets compressed and travels each time
  • writing/updating occasions read and write costs within the sync stored procedure each time
  • competing writers pay all but the update case in the case of a re-run of the same transaction
  • readers not interested in/unable to process a specific snapshot schema still have to pay the price to transport it
  • each update goes to the changefeed triggering more read costs
  • the size and cost of the snapshots is harder to separate out
  • snapshots contribute to partition size limits

Thus it makes sense to do some, or all of the following in the case where there is a projector in play and there is an aux collection (in the case of multiple writers one might even consider making an aux collection even if you don't have a projector):

  • maintain the snapshots in the aux collection, where one is not paying for changefeed re-reads of the same data
  • reduce the write frequency for snapshots (when the write happens should be deterministic for competing writers)
  • implement ability for stored proc to maintain events since snapshot in tip when not updating snapshot
  • write to a nominated collection (typically aux), with a deterministic guid based on the case / event type tag of the union as a point write
  • refer to the snapshot in the unfolds; automate loading of the referenced snapshot if picked by isOrigin
  • expose the ability to customize the snapshot loading process (but batteries should be included)
    • (perhaps do a sample showing parallel reads of the Snapshots and the Tip?)

bartelink avatar Dec 08 '18 10:12 bartelink

The designs for #145 and #157 should consider the potential overlap with this requirement.

bartelink avatar Sep 04 '19 09:09 bartelink