repmgr
repmgr copied to clipboard
EnterpriseDB
too many forks
Greetings
We have a 2 node setup. When the primary node goes offline (eg stop postgres & repmgr), the standby would promote itself. The setup used to work - I am not aware of any changes. The problem I see is that the promote command is forked until the max connections (set to 100) of the database are exhausted and accepts no more connections;
[..]
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [ERROR] connection to database failed
[2021-09-10 16:57:56] [DETAIL]
FATAL: sorry, too many clients already
[..]
When running "ps -ef" immediately when the promote is executed, there are indeed many "standby promote" commands running
Versions: $ rpm -qa|egrep 'postgres|repmgr' postgresql11-11.8-1PGDG.rhel7.x86_64 repmgr11-5.2.1-1.el7.x86_64 postgresql11-contrib-11.8-1PGDG.rhel7.x86_64 postgresql11-libs-11.8-1PGDG.rhel7.x86_64 postgresql11-server-11.8-1PGDG.rhel7.x86_64
Thanks in advance
repmgr.conf
node_id= 2 # A unique integer greater than zero
node_name= node-b # An arbitrary (but unique) string; we recommend
conninfo= 'host=10.222.67.87 user=repmgr dbname=repmgr' # Database connection information as a conninfo string.
data_directory= '/var/lib/pgsql/11/data' # The node's data directory. This is needed by repmgr
replication_user= postgres # User to make replication connections with, if not set
location= default # An arbitrary string defining the location of the node; this
use_replication_slots= 1 # whether to use physical replication slots
log_level= INFO # Log level: possible values are DEBUG, INFO, NOTICE,
log_facility= STDERR # Logging facility: possible values are STDERR, or for
log_file= '/var/log/repmgr/repmgr.log' # STDERR can be redirected to an arbitrary file
log_status_interval= 300 # interval (in seconds) for repmgrd to log a status message
pg_bindir= '/usr/pgsql-11/bin' # Path to PostgreSQL binary directory (location
repmgr_bindir= '/usr/pgsql-11/bin' # Path to repmgr binary directory (location of the repmgr
pg_basebackup_options= '--wal-method=fetch' # Options to append to "pg_basebackup"
ssh_options= '-q -o ConnectTimeout=10' # Options to append to "ssh"
failover= automatic # one of 'automatic', 'manual'.
priority= 100 # indicates a preferred priority for promoting nodes;
connection_check_type= ping # How to check availability of the upstream node; valid options:
promote_command= '/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file' # command repmgrd executes when promoting a new primary; use something like:
follow_command= '/usr/pgsql-11/bin/repmgr standby follow -f /var/lib/pgsql/11/repmgr.conf --upstream-node-id=%n --log-to-file' # command repmgrd executes when instructing a standby to follow a new primary;
monitoring_history= 'True' # Whether to write monitoring data to the "montoring_history" table
repmgrd_pid_file= '/var/lib/pgsql/11' # Path of PID file to use for repmgrd; if not set, a PID file will
service_start_command = 'systemctl start postgresql-11'
service_stop_command = 'systemctl stop postgresql-11'
service_restart_command = 'systemctl restart postgresql-11'
service_reload_command = 'systemctl reload postgresql-11'
service_promote_command = '/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file' # This parameter is intended for systems which provide a
repmgrd_service_start_command = 'systemctl start repmgr'
repmgrd_service_stop_command = 'systemctl stop repmgr'
full log
[2021-09-10 16:57:05] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:57:05] [DETAIL] PQping() returned "PQPING_REJECT"
[2021-09-10 16:57:05] [WARNING] unable to connect to upstream node "node-a" (ID: 1)
[2021-09-10 16:57:05] [INFO] checking state of node "node-a" (ID: 1), 1 of 6 attempts
[2021-09-10 16:57:05] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:05] [DETAIL] PQping() returned "PQPING_REJECT"
[2021-09-10 16:57:05] [INFO] sleeping up to 10 seconds until next reconnection attempt
[2021-09-10 16:57:15] [INFO] checking state of node "node-a" (ID: 1), 2 of 6 attempts
[2021-09-10 16:57:15] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:15] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:15] [INFO] sleeping up to 10 seconds until next reconnection attempt
[2021-09-10 16:57:25] [INFO] checking state of node "node-a" (ID: 1), 3 of 6 attempts
[2021-09-10 16:57:25] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:25] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:25] [INFO] sleeping up to 10 seconds until next reconnection attempt
[2021-09-10 16:57:35] [INFO] checking state of node "node-a" (ID: 1), 4 of 6 attempts
[2021-09-10 16:57:35] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:35] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:35] [INFO] sleeping up to 10 seconds until next reconnection attempt
[2021-09-10 16:57:45] [INFO] checking state of node "node-a" (ID: 1), 5 of 6 attempts
[2021-09-10 16:57:45] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:45] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:45] [INFO] sleeping up to 10 seconds until next reconnection attempt
[2021-09-10 16:57:55] [INFO] checking state of node "node-a" (ID: 1), 6 of 6 attempts
[2021-09-10 16:57:55] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:55] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:55] [WARNING] unable to reconnect to node "node-a" (ID: 1) after 6 attempts
[2021-09-10 16:57:55] [INFO] 0 active sibling nodes registered
[2021-09-10 16:57:55] [INFO] 2 total nodes registered
[2021-09-10 16:57:55] [INFO] primary node "node-a" (ID: 1) and this node have the same location ("default")
[2021-09-10 16:57:55] [INFO] no other sibling nodes - we win by default
[2021-09-10 16:57:55] [NOTICE] this node is the only available candidate and will now promote itself
[2021-09-10 16:57:55] [INFO] promote_command is:
"/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"
[2021-09-10 16:57:56] [ERROR] connection to database failed
[2021-09-10 16:57:56] [DETAIL]
FATAL: sorry, too many clients already
[2021-09-10 16:57:56] [DETAIL] attempted to connect using:
user=repmgr dbname=repmgr host=10.222.67.87 connect_timeout=2 fallback_application_name=repmgr options=-csearch_path=
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] promote command failed
[2021-09-10 16:57:56] [DETAIL] promote command exited with error code 8
[2021-09-10 16:57:56] [INFO] checking if original primary node has reappeared
[2021-09-10 16:57:56] [INFO] searching for primary node
[2021-09-10 16:57:56] [INFO] checking if node 1 is primary
[2021-09-10 16:57:56] [ERROR] connection to database failed
[2021-09-10 16:57:56] [DETAIL]
could not connect to server: Connection refused
Is the server running on host "10.222.67.86" and accepting
TCP/IP connections on port 5432?
[2021-09-10 16:57:56] [DETAIL] attempted to connect using:
user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr options=-csearch_path=
[2021-09-10 16:57:56] [INFO] checking if node 2 is primary
[2021-09-10 16:57:56] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:57:56] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:56] [ERROR] unable to execute get_primary_current_lsn()
[2021-09-10 16:57:56] [DETAIL]
FATAL: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
[2021-09-10 16:57:56] [WARNING] unable to retrieve primary's current LSN
[2021-09-10 16:57:58] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:57:58] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:58] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:57:58] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:58] [WARNING] monitoring_history requested but primary connection not available
[2021-09-10 16:58:00] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:58:00] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:58:00] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:58:00] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:58:00] [WARNING] monitoring_history requested but primary connection not available
[2021-09-10 16:58:02] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:58:02] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
You have this entry in your repmgr.conf file:
service_promote_command = '/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file'
The full comment in the sample repmgr.conf file (which is partially included in your repmgr.conf file) is:
#service_promote_command = '' # This parameter is intended for systems which provide a
# package-level promote command, such as Debian's
# "pg_ctlcluster". *IMPORTANT*: it is *not* a substitute
# for "promote_command"; do not use "repmgr standby promote"
# (or a script which executes "repmgr standby promote") here.
so what's happening is that repmgr standby promote is executing the command defined in service_promote_command, which is repmgr standby promote, which will continue executing itself until resources are exhausted.
So just remove the service_promote_command.
