pgsink icon indicating copy to clipboard operation
pgsink copied to clipboard

Optimise successive imports (for quick removal, re-add of tables)

Open lawrencejones opened this issue 4 years ago • 0 comments

At present, removal of a table from the publication causes us to consider that table totally unreliable. In order to re-add the table to our stream, we'd schedule an import of all existing data, in order to ensure we haven't missed anything.

Given users should be able to remove tables without too much worry, especially if one table has changes that are incompatible with the sink, imports should attempt to be cheap. If we were to track the LSN at which we ceased to stream changes, we might be able to filter which rows are included in a subsequent import by using the xmin field on each row.

lawrencejones avatar May 04 '20 18:05 lawrencejones