pgsync Synchronization is not matching

PGSync version: 3.0.0

Postgres version: PostgreSQL 15 Aurora RDS

Elasticsearch/OpenSearch version: OpenSearch v 2.9.0

Redis version: LATEST

Python version: 3.10.12

Problem Description: PGSYNC is being run every 5 minutes with the "pgsync" command.

It works very well, however after a while the synchronization does not match. There is usually a difference of 1 or 2 records.

This difference in records happens after a few days.

And when that happens, I have to reindex all the records again.

this happens with both -deamon and --polling.

environment:

ELASTICSEARCH_STREAMING_BULK=True ELASTICSEARCH=False OPENSEARCH=True SCHEMA=schema.json CHECKPOINT_PATH=checkpoint ELASTICSEARCH_TIMEOUT=20000

Dec 22 '23 05:12 leokaynan-bayer

One thing to mention, the state file is very important This is checkpoint file starts with a .<db_name>_<index_name> are you by any chance deleting this file/is this checkpoint path always accessible?

Dec 22 '23 16:12 toluaina

One thing to mention, the state file is very important This is checkpoint file starts with a .<db_name>_<index_name> are you by any chance deleting this file/is this checkpoint path always accessible?

The checkpoint file has not been deleted or changed.

The difference in records is random.

And when it happens, it shows that there is no data to be synchronized.

Dec 27 '23 00:12 leokaynan-bayer