paimon
paimon copied to clipboard
[FLINK-31462] Supports full calculation from the specified snapshot in streaming mode
Purpose
When consuming multiple tables, it may be necessary to align data versions in different tables to ensure version consistency. However, the current stream mode only supports incremental processing from the specified snapshot, and does not support full calculation of the specified snapshot first, and then incremental calculation.
Brief change
A new startup mode FROM_SNAPSHOT_FULL is added to support full calculation of the specified snapshot of data in streaming mode, and then incrementally process the newly added data to ensure the consistency of the data version when processing multiple tables.