Backups can fail silently - Failed backup notifications are not being sent

[WadeBarnes]

The latest version of the backup-container appears to be failing to send failed backup notifications.

We encountered an issue where the backup volume filled up and was causing backups to fail.

Backing up 'wallet-idim/agent_idim_wallet' to '/backups/daily/2025-02-03/wallet-idim-agent_idim_wallet_2025-02-03_01-00-00.sql.gz.in_progress' ...

gzip: stdout: No space left on device
removed '/backups/daily/2025-02-03/wallet-idim-agent_idim_wallet_2025-02-03_01-00-00.sql.gz.in_progress'
[!!ERROR!!] - Failed to backup postgres=wallet-idim/agent_idim_wallet.

Elapsed time: 0h:0m:56s - Status Code: 141

The notifications for these backup failures was never sent, however the subsequent verification of a backup that was corrupted by this issue did cause a failed verification notification to be sent.

This means backups can fail silently and if there is no subsequent failed verification notification the issue can go unnoticed.