rfcs icon indicating copy to clipboard operation
rfcs copied to clipboard

Published 20 hours ago • verified publisher icon risingwavelabs

RFC: decouple sink and checkpoint

Open wenym1 opened this issue 2 years ago • 2 comments

A rendered link

wenym1 avatar Feb 27 '23 08:02 wenym1

We had a meeting last Friday. Here is some summaries:

  • For a quick and simple implementation, we can implement a log store via kv interface first.
  • Sinks can asynchronously consume and commit the data. Only the unconsumed data will be flushed to persistent storage for consistency of checkpoint. Sink can report consume progress to help deleting the consumed data.

wenym1 avatar Feb 27 '23 08:02 wenym1

Is this finished (and can be merged)?

xxchan avatar Oct 26 '23 08:10 xxchan