pgbackrest icon indicating copy to clipboard operation
pgbackrest copied to clipboard

Backup command end: terminated on signal [SIGHUP]

Open jasonchenTJ opened this issue 1 year ago • 2 comments

Please provide the following information when submitting an issue (feature requests or general comments can skip this):

  1. pgBackRest version: pgBackRest 2.50

  2. PostgreSQL version: pg16.2

  3. Operating system/version - if you have more than one server (for example, a database server, a repository host server, one or more standbys), please specify each:

  4. Did you install pgBackRest from source or from a package? From source

  5. Please attach the following as applicable:

    • pgbackrest.conf file(s)
    • postgresql.conf settings applicable to pgBackRest (archive_command, archive_mode, listen_addresses, max_wal_senders, wal_level, port)

postgres=# show archive_command; archive_command

pgbackrest --stanza=prod5007 archive-push %p (1 row)

postgres=# show archive_mode; archive_mode

on (1 row)

postgres=# show listen_addresses; listen_addresses

(1 row)

postgres=# show max_wal_senders; max_wal_senders

10 (1 row)

postgres=# show wal_level; wal_level

logical (1 row)

postgres=# show port; port

5007 (1 row)

- errors in the postgresql log file before or during the time you experienced the issue
- log file in `/var/log/pgbackrest` for the commands run (e.g. `/var/log/pgbackrest/mystanza_backup.log`)
  1. Describe the issue: prod5007-backup.log

backup_full.log

Full backup process of 10TB size database with 16 parallel, it was stopped with error 'terminated on signal [SIGHUP]'

jasonchenTJ avatar Apr 29 '24 02:04 jasonchenTJ

It's a remote backup. I suspect this error due to network unstable.

jasonchenTJ avatar Apr 29 '24 23:04 jasonchenTJ

Looks like the terminal session that was running pgbackrest got terminated. If it is not running under cron you might try using screen.

dwsteele avatar Apr 30 '24 08:04 dwsteele

@dwsteele Thanks for your feedback!

I resolved it as below:

Increase the timeout for SSH remote connection:

sudo vi /etc/ssh/sshd_config

modify below parameters: #ClientAliveInterval default is 0 seconds #ClientAliveCountMax default is 3

I modified as

ClientAliveInterval 20 ClientAliveCountMax 3

Besides, also need to reload the conf:

sudo systemctl reload sshd

======================================== So it that possible add a custom parameter for the SSH timeout on pgbackrest conf file ?

like command: ssh -o ConnectTimeout=10

Thanks Jason

jasonchenTJ avatar May 06 '24 01:05 jasonchenTJ

So it that possible add a custom parameter for the SSH timeout on pgbackrest conf file

No, but you should be able to set whatever client settings you need in ~/.ssh/ssh_config.

dwsteele avatar May 09 '24 01:05 dwsteele

@dwsteele Thank you! will follow your suggestion.

jasonchenTJ avatar May 09 '24 01:05 jasonchenTJ