Backups stopped working

[arezk84]

trafficstars

Environment

Please provide the following details:

• Platform: (Kubernetes)
• Platform Version: (4.7.4)
• PGO Image Tag: (centos8-13.5-4.7.4)
• Postgres Version (13)
• Storage: (oci Orcale cloud)

I created a cluster and scheduled a backup every 1 hour, after ~1 month I found backups stopped and I found these errors in the database log

repo1: [FileWriteError] raised from remote-0 protocol on 'productiondatabase-backrest-shared-repo': unable to write to fd: [28] No space left on device
2022-07-08 00:48:26.133 UTC [4895] LOG:  archive command failed with exit code 103
2022-07-08 00:48:26.133 UTC [4895] DETAIL:  The failed archive command was: source /opt/crunchy/bin/postgres-ha/pgbackrest/pgbackrest-set-env.sh && pgbackrest archive-push "pg_wal/00000007.history"
2022-07-08 00:48:26.133 UTC [4895] WARNING:  archiving write-ahead log file "00000007.history" failed too many times, will try again later

I expect that I reached the size limit of backup volume but when I checked the space of volume I found it still has space.

PVC                         INSTANCE                               POD                                                     TYPE        USED      CAPACITY  % USED
--------------------------- -------------------------------------- ------------------------------------------------------- ----------- --------- --------- ------
productiondatabase           productiondatabase                      productiondatabase-755cb6d8b7-cc7s9                      data        2GiB      128GiB    1%
productiondatabase-pgbr-repo productiondatabase-backrest-shared-repo productiondatabase-backrest-shared-repo-6f5fb5cd4f-r8kgp pgbackrest  8GiB      256GiB    3%

Any idea what should I do? Thanks

[tazhate]

No space left on device

Add space to backrest pvc.

[arezk84]

@tazhate I did. nothing has changed. what I did is restarted the deployment of the database and it seems fixed

[tazhate]

@arezk84 after you edited pvc it doesnt add space automatically. pod restart is required in most cases.

[tjmoore4]

Since this issue involves functionality from an older version of Crunchy Postgres for Kubernetes, I am proceeding with closing this issue. If you have any additional questions - or if you would like to discuss scenarios, processes, functionality, etc. with Crunchy Postgres for Kubernetes v5 - further, please feel free to reach out on the PostgreSQL Operator community mailing list. And for more information about Crunchy Postgres for Kubernetes v5 (including guidance for upgrading from v4), please see the latest documentation:

https://access.crunchydata.com/documentation/postgres-operator/latest/

Thanks!