hydroflow icon indicating copy to clipboard operation
hydroflow copied to clipboard

Checkpoint/migration handling

Open MingweiSamuel opened this issue 1 year ago • 1 comments

Split out of #929

Let's get all the state of a transducer exposed for management including checkpoint/restart.

  • What about state that persists across executions (e.g. in storage or a replication library)?
  • Relates to the "tables" in Bloom being absent from Hydroflow: is state first-class in the IR and separate from ops?
    • Might be good to have an explicit data model spec

from @jhellerstein

MingweiSamuel avatar Jan 29 '24 17:01 MingweiSamuel

Arguably our philosophy could be that the Hydroflow runtime itself does not provide intrinsic mechanisms for efficient checkpoint. Instead we defer checkpoint to a system either beneath Hydroflow, or built out of Hydroflow.

jhellerstein avatar Mar 25 '24 16:03 jhellerstein