toluaina
pgsync on daemon mode dont insert into opensearch
PGSync version: 2.5.0
Postgres version: Aurora 14.5
Elasticsearch version: Opensearch 2.3
Redis version: 5.0.7
Python version: 3.8.10
Problem Description: If i run pgsync everything works, i get the data inserted into opensearch, but if i try to run it using the --daemon flag, pgsync detect the changes on the rds, but it never insert them into opensearch
the only thing i see is:
Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]... Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]... Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]... Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]...
i tried deleting the construct file and start the process again, still same issue.
i also followed this steps:
Did you run bootstrap successfully? did you alter the structure of the schema.json There is a checkpoint file which maintains the state. Try deleting .construct_pg_posts and rerun
if i restart pgsync on deamon mode, it inserts the rows that were pending from the previous run, but then any new row is not inserted.
the schema definition im using is:
[
{ "database": "client_facing", "index": "blue-sky-rules-schema", "nodes": { "table": "v_security_stats", "schema": "ref_data", "columns": [ "source", "status", "secid", "comp_id", "cusip", "issuer_name", "symbol" ], "primary_key": ["secid"] } } ]
Error Message (if any):
My guess is this has to do with AWS Aurora. PGSync has customs modifications for AWS RDS.
can you please run this query on your db
SELECT name FROM pg_settings ORDER BY name
