CrunchyData
Bucket retention policy deleted archive.info
Overview
At some point backups just stopped working for one of our databases and pgBackRest team doesn't think it's related to them: https://github.com/pgbackrest/pgbackrest/issues/1905
registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:centos8-2.35-0
registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-13.4-1
Sorry for the cross post, but it seems to be the interaction between these two products that needs solving, and in any case, backups just starting to fail is a bug.
Environment
Please provide the following details:
- Platform: (
Kubernetes,OpenShift,Rancher,GKE,EKS,AKSetc.) GKE / GCS - Platform Version: (e.g.
1.20.3,4.7.0) - PGO Image Tag: (e.g.
ubi8-5.2.0-0) - Postgres Version (e.g.
14) 13.4 - Storage: (e.g.
hostpath,nfs, or the name of your storage class) gcs
Steps to Reproduce
REPRO
Let a low-traffic cluster run alone for a year with misc pod restarts due to upgrades, of course.
EXPECTED
Not to have archive.info files just "disappearing".
ACTUAL
It seems to have disappeared
Logs
https://github.com/pgbackrest/pgbackrest/issues/1905
So after discussions in the other thread, it would seem the bucket retention policy deleted the file.
How do I make postgres-operator reinitialise the stanza?
When creating a new repo:
time="2022-10-21T07:51:21Z" level=error msg="unable to create stanza" error="command terminated with exit code 47: ERROR: [047]: unable to create path '/var/lib/pgbackrest/archive': [30] Read-only file system\n" file="internal/controller/postgrescluster/pgbackrest.go:2580" func="postgrescluster.(*Reconciler).reconcileStanzaCreate" name=analytics namespace=db reconciler=pgBackRest reconciler group=postgres-operator.crunchydata.com reconciler kind=PostgresCluster version=5.1.2-0
@haf just wanted to see if you are still running into trouble stanza creation and/or anything having to do with your cloud backups. From what I can see here, a misconfiguration was the likely culprit for the stanza create errors.
Considering the age of this issue I am going to proceed with closing it, but feel free to reach out if you still have any additional questions or issues. I'll note that you can now reach out via the PGO project community discord server as well.
