backup-container icon indicating copy to clipboard operation
backup-container copied to clipboard

Add retry to the backup process

Open WadeBarnes opened this issue 5 years ago • 0 comments

Occasionally a backup can run into an issue and fail on the first attempt. If attempted again a sort time later it will succeed.

For the sake of robustness add a configurable retry and delay to the backup process so backups are retried if they fail the first time.

Example backup failure:

Backing up 'wallet-indy-cat:5432/agent_indy_cat_wallet' to '/backups/daily/2020-08-22/wallet-indy-cat-agent_indy_cat_wallet_2020-08-22_01-05-02.sql.gz.in_progress' ...
--
  | pg_dump: Dumping the contents of table "items" failed: PQgetCopyData() failed.
  | pg_dump: Error message from server: server closed the connection unexpectedly
  | This probably means the server terminated abnormally
  | before or while processing the request.
  | pg_dump: The command was: COPY public.items (id, type, name, value, key) TO stdout;
  | removed '/backups/daily/2020-08-22/wallet-indy-cat-agent_indy_cat_wallet_2020-08-22_01-05-02.sql.gz.in_progress'
  | [!!ERROR!!] - Failed to backup wallet-indy-cat:5432/agent_indy_cat_wallet.
  |  
  | Elapsed time: 0h:25m:53s - Status Code: 1

The database server did not terminate. When the backup was retried it succeeded.

WadeBarnes avatar Aug 24 '20 15:08 WadeBarnes