repmgr icon indicating copy to clipboard operation
repmgr copied to clipboard

Upstream connection: CRITICAL witness is not attached to expected upstream node

Open mattbunter opened this issue 1 year ago • 0 comments

Repmgr 5.1 Postgres 12 CentOS 7.9.2009 (Core)

After two switchovers, node 1 to node 2 then node 2 to node 1, the witness is reporting an error :

Upstream connection: CRITICAL (node "pgwitness.themis.uk.azure" (ID: 577739013) is not attached to expected upstream node "pgsql1.themis.uk.azure" (ID: 577714693))

Tried forcing register and also unregister then register :

[postgres@pgwitness ~]$ repmgr -f /etc/repmgr/12/repmgr.conf witness unregister --dry-run INFO: connecting to node "pgwitness.themis.uk.azure" (ID: 577739013) INFO: prerequisites for unregistering the witness node are met [postgres@pgwitness ~]$ repmgr -f /etc/repmgr/12/repmgr.conf witness unregister INFO: connecting to node "pgwitness.themis.uk.azure" (ID: 577739013) INFO: unregistering witness node 577739013 INFO: witness unregistration complete DETAIL: witness node with ID 577739013 successfully unregistered [postgres@pgwitness ~]$ [postgres@pgwitness ~]$ [postgres@pgwitness ~]$ repmgr -f /etc/repmgr/12/repmgr.conf witness register -h pgsql1.themis.uk.azure INFO: connecting to witness node "pgwitness.themis.uk.azure" (ID: 577739013) INFO: connecting to primary node INFO: "repmgr" extension is already installed ERROR: witness node is already initialised and contains node records HINT: use option -F/--force to reinitialise the node [postgres@pgwitness ~]$ repmgr -f /etc/repmgr/12/repmgr.conf witness register -F -h pgsql1.themis.uk.azure INFO: connecting to witness node "pgwitness.themis.uk.azure" (ID: 577739013) INFO: connecting to primary node INFO: "repmgr" extension is already installed INFO: witness registration complete NOTICE: witness node "pgwitness.themis.uk.azure" (ID: 577739013) successfully registered [postgres@pgwitness ~]$ node_check Node "pgwitness.themis.uk.azure": Server role: OK (node is witness) Replication lag: OK (N/A - node is witness) WAL archiving: OK (0 pending archive ready files) Upstream connection: CRITICAL (node "pgwitness.themis.uk.azure" (ID: 577739013) is not attached to expected upstream node "pgsql1.themis.uk.azure" (ID: 577714693)) Downstream servers: OK (this node has no downstream nodes) Replication slots: OK (node has no physical replication slots) Missing physical replication slots: OK (node has no missing physical replication slots) Configured data directory: OK (configured "data_directory" is "/data/12") [postgres@pgwitness ~]$ node_status Node "pgwitness.themis.uk.azure": PostgreSQL version: 12.2 Total data size: 32 MB Conninfo: host=10.94.5.99 user=repmgr dbname=repmgr Role: witness WAL archiving: enabled Archive command: /bin/true WALs pending archiving: 0 pending files Replication connections: 0 (of maximal 10) Replication slots: 0 physical (of maximal 10; 0 missing) Replication lag: n/a

Not sure what to do now. Would appreciate advice. Unable to upgrade repmgr at this time.

mattbunter avatar Sep 04 '23 11:09 mattbunter