rfcs icon indicating copy to clipboard operation
rfcs copied to clipboard

RFC: Support sink coordinator

Open wenym1 opened this issue 2 years ago • 1 comments
trafficstars

rendered file: https://github.com/risingwavelabs/rfcs/blob/yiming/sink-coordinator/rfcs/0061-sink-coordinator.md

wenym1 avatar May 16 '23 22:05 wenym1

Summary for the design meeting:

  • In sink coordinator design, there will be multiple concurrent sink writer in each sink parallelism (single sink executor) and a single sink coordinator in the meta store.
  • When checkpoint barrier comes, we call finish_write on all sink writers and get the metadata written by each sink writer. After collecting metadata of all sink writers, the coordinator will commit all metadata to the external sinks.
  • We will implement a simple stateless coordinator for improving the throughput of iceberg sink at first.
  • The implementation logic of two phase commit sink is correct, but complex compared to the simple stateless coordinator. Will implement it when there is more feature demand on exactly-once sink.

wenym1 avatar May 24 '23 04:05 wenym1