burr icon indicating copy to clipboard operation
burr copied to clipboard

Transactional/commit-based state

Open elijahbenizzy opened this issue 1 year ago • 1 comments

The plan here is from the docs. Problem is that we're eagerly evaluating state at every point which is not particularly efficient (TBD, however). This also helps us solve this in a clean way: https://github.com/DAGWorks-Inc/burr/issues/28.

We plan the ability to manage state in a few ways:

  1. commit -- an internal tool to commit/compile a series of changes so that we have the latest state evaluated
  2. persist -- a user-facing API to persist state to a database. This will be pluggable by the user, and we will have a few built-in options (e.g. a simple in-memory store, a file store, a database store, etc...)
  3. hydrate -- a static method to hydrate state from a database. This will be pluggable by the user, and we will have a few built-in options that mirror those in persist options.

Currently state is immutable, but it utilizes an inefficient copy mechanism. This is out of expedience -- we don't anticipate this will be painful for the time being, but plan to build a more efficient functional paradigm. We will likely have:

  1. Each state object be a node in a linked list, with a pointer to the previous state. It carries a diff of the changes from the previous state.
  2. An ability to checkpoint (allowing for state garbage collection), and store state in memory/kill out the pointers.

We will also consider having the ability to have a state solely backed by redis (and not memory), but we are still thinking through the API.

elijahbenizzy avatar Feb 25 '24 19:02 elijahbenizzy

Reading for a conceptual model https://martinfowler.com/eaaDev/EventSourcing.html

elijahbenizzy avatar Apr 10 '24 15:04 elijahbenizzy