pgsync
pgsync copied to clipboard
toluaina
New documents not being created on database insert
Hello, I am running the docker version of the airbnb demo. When I insert new records into postgres, I am not seeing the elastic search document count increase. I do however see "Polling db airbnb: 2 item(s)" logged
.
Also, are there instructions on how to setup pgsync for an existing database somewhere? I want to continue running pgsync in docker, but have it work with a db running on bare metal on my local Mac.
Thanks Lane
I think the getting started section might be the best things we have. https://pgsync.com/getting-started. I can create a section specifically for setting up with an existing db.
The logs suggest it is pushing to Elasticsearch. Do you see any error previously?
@toluaina , even I am facing the same Issue. New records are not going to ES, but it is showing polling count correctly in PGSync logs. Also I noticed that if we restart PGSync all the records sync to ES correctly. But its not ideal to restart it everytime.
Data is first held in the replication logs and sometimes this needs to be processed first. I have tried to improve the logging to indicate what pgsync is doing at each stage. My guess is that the initial sync is not yet complete. You can see how this works better with a smaller dataset.
@toluaina
i did initial sync via bootstrap -c schema.json command and it's get done successfully now after a month new records not going to ES, but it's showing pulling count currently in PGSync logs. whenever I do pgsync -c schema.json it will pull new records to ES.
In which direction should I look? Is there something wrong with the Replication slot? What is the minimum replication slot required?
Is there a way we can make sure that all the data are get indexed and flush the logs from the replica slot and reset pgsync pulling count to 0 in files?
@AnkitKundariya The latest version on master should show you more details in the console. Can you post the output here? Also add/update a record in the db and post the result from the colsole
@toluaina I have my schema with the following details:
If i am trying to edit existing user, its reflecting the changes in Elasticsearch but its not reflecting new user entry in Elasticsearch. Delete is also working.
