pgsync icon indicating copy to clipboard operation
pgsync copied to clipboard

when using through_tabes, New data added in Postgres is not getting sync to elasticsearch

Open 635503281 opened this issue 4 years ago • 7 comments

PGSync version: 2.1.1

Postgres version: 13.2

Elasticsearch version: 7.6

Redis version: 6.2.2

Python version: 3.7.10

Problem Description:

The initial synchronization of pgsync was successful. The added data was not synchronized to the ES, and there was no error. When I used comment as only children, it synchronize success. I can now add a channel child and use through_tables is associated with tables. When adding data, it is not synchronized to the ES, and there is no error message. I restart pgsync, and it can synchronize all, but it is still not synchronized to the es when adding data; I tried to associate only one children in the channel, but failed to add data. It can be seen use through_tables as associated, the synchronization is unsuccessful and no error is reported. schema.json is as follows: image

Database table posts_channel is an association table automatically created using orm image Please look at this problem. It's urgent.

Error Message (if any): After adding data, no error is reported, but the data is not synchronized to es. image

635503281 avatar Aug 23 '21 02:08 635503281

I'm having somewhat of a similar issue.

If I add a relationship with "through_tables" the sync will not run, it just lists out the different table names and creates the hidden checkpoint file. The minute I remove the child with the through_tables value it works fine.

I've deleted the index, re-run the bootstrap and deleted the checkpoint file, but it will not sync the data unless I remove the through_tables realationship.

selected-pixel-jameson avatar Aug 25 '21 12:08 selected-pixel-jameson

  • What do you mean by the sync will not run.
  • Looks to me like something is wrong with the way you have defined the through_table
  • If you change the structure of the schema.json you will need to remove the checkpoint file and re-index.
  • Do you have a foreign key between the posts_channel correctly.
  • If this is urgent please provide your database schema dump (no-data) and I will try to reproduce.

toluaina avatar Aug 27 '21 09:08 toluaina

After adding through_table, adding new data is not synchronized to es, restarting pgsync will synchronize all the data in the database to es. I deleted the index in es, and the posts_channel foreign key is correct. I currently use the one-to-many association posts_channel method. image To achieve the purpose by querying the fields in the associated posts_channel

635503281 avatar Aug 27 '21 10:08 635503281

Adding an explicit foreign_key to your schema implies that you don't have the foreign key constraints defined in your database. I would still like to see your database schema to get a better understanding.

Your first schema did not have foreign_key but this one does. Adding foreign_key in the relationship is required ONLY when your database does do define this relationship between the tables.

toluaina avatar Aug 27 '21 18:08 toluaina

I am facing the same issue for one-to-many relationships where adding records are not being synced but edit and deletes are working fine. Are there any updates on this?

Dipesh-Aspire avatar Mar 08 '22 15:03 Dipesh-Aspire

@Dipesh-Aspire Can you open a new request with some details please

toluaina avatar Mar 08 '22 22:03 toluaina

@toluaina It seems I got the similar thread here: https://github.com/toluaina/pgsync/issues/182

posting my details there.

Dipesh-Aspire avatar Mar 09 '22 07:03 Dipesh-Aspire