citus icon indicating copy to clipboard operation
citus copied to clipboard

Consider adding a history of topology changes

Open marcocitus opened this issue 2 years ago • 0 comments

To implement CDC for Citus, we would need to implement a logical decoder that filters out inserts that happen as a result of internal data moves (e.g. shard move, shard split, create_distributed_table, alter_distributed_table, undistribute_table).

The logical decoder may be arbitrarily far behind, and the state of the cluster at the time of the insert it is currently decoding may have been very different. Therefore, to know whether an insert needs to be decoded, it would be useful if Citus kept a history of topology changes. That way, the decoder can reconstruct the relevant metadata at a given LSN from the history. In particular, the history would tell us at which LSN a particular shard became active, and which distributed table (name) it was part of.

Keeping a history would also be useful for users to understand how their cluster evolved over time, and be able to debug issues that happened in the past.

marcocitus avatar Jul 13 '22 08:07 marcocitus