DB archival can cause slowness in full reload over time

[justjkk]

Multiple tables like communicator.genericsms archive data older than a few months to S3 in order to save space. Over time this can cause more entries in Kafka that slows down a full reload. Need to figure out a way to periodically refresh the Debezium data so that data is stored as snapshots rather than events.

[justjkk]

This issue can also happen due to some tables that are doing a lot of updates like sessions table last_accessed_at column.

[alok87]

Log compaction can be used here. It can help keep only the latest record for key. @justjkk