rfcs
rfcs copied to clipboard
RFC: Support sink coordinator
trafficstars
rendered file: https://github.com/risingwavelabs/rfcs/blob/yiming/sink-coordinator/rfcs/0061-sink-coordinator.md
Summary for the design meeting:
- In sink coordinator design, there will be multiple concurrent sink writer in each sink parallelism (single sink executor) and a single sink coordinator in the meta store.
- When checkpoint barrier comes, we call
finish_writeon all sink writers and get the metadata written by each sink writer. After collecting metadata of all sink writers, the coordinator will commit all metadata to the external sinks. - We will implement a simple stateless coordinator for improving the throughput of iceberg sink at first.
- The implementation logic of two phase commit sink is correct, but complex compared to the simple stateless coordinator. Will implement it when there is more feature demand on exactly-once sink.