b4mad
pgo storage fills up too quick / different psql operator?
because backups are eating up our storage: https://github.com/CrunchyData/postgres-operator/issues/2531
Consider migration to https://cloudnative-pg.io/
https://blog.palark.com/cloudnativepg-and-other-kubernetes-operators-for-postgresql/
either increasing the backup frequency might help as in https://github.com/CrunchyData/postgres-operator/issues/2531#issuecomment-922349084
or setting archive_mode to off as in https://github.com/CrunchyData/postgres-operator/issues/2531#issuecomment-1022070211
Extend the PVC for psql by +2Gb and wait until the DB recovers. Then trigger a one off backup either by:
oc annotate postgrescluster db --overwrite postgres-operator.crunchydata.com/pgbackrest-backup="$(date)"
or oc create job --from=cronjob/db-repo1-full db-backup-full
Long term we need either
- monitoring of the psql PVC (but this happens fast)
- a setup that can tolerate failing backups
- less WAL for psql
implemented some altering, see https://github.com/b4mad/racing/blob/main/manifests/env/phobos/postgresql/alerting.yaml
alertmanager will send to https://discord.com/channels/850841654327640074/1106466539273732136