UnableToCreateStanzas - S3 Backup Schedule

[B1ue-W01f]

Overview

Following the S3 backup example install and adding the example schedule, deployed successfully with all resources successful.

Connection to S3 is successful for backup with meta data files being written across backup and archive directories, however data backup does not occur.

Error on PostgreCluster reporting as:

command terminated with exit code 47: ERROR: [047]: unable to create path '/var/lib/pgbackrest/archive': [30] Read-only file system 

Environment

• Platform: Kubernetes
• Platform Version: 1.24.1
• PGO Image Tag: ubi8-14.3-0
• Postgres Version: 14
• Storage: rook-ceph + S3 for backup

Steps to Reproduce

REPRO

Provide steps to get to the error condition:

1. Install using S3 example: https://access.crunchydata.com/documentation/postgres-operator/v5/quickstart/
2. Configure S3 storage

EXPECTED

Backup completes or error message indicates resolution

ACTUAL

Backup starts but dumb isn't generated

Logs

command terminated with exit code 47: ERROR: [047]: unable to create path '/var/lib/pgbackrest/archive': [30] Read-only file system

Additional Information

Ive seen mention that since 4.7 some images are now read only, which appears to correlate with the error I'm receiving (presumably that an export path (/var/lib/pgbackrest/archive) isn't writable. I can't find any information within the guidance related to additional volumes though so presume this isn't the issue?

[B1ue-W01f]

Appears to relate to comments in #2559

[B1ue-W01f]

Interestingly going through the narrative on here, which seems a fair approach: https://www.crunchydata.com/blog/secure-permissions-for-pgbackrest Comparatively, with the image being set to read only, and seemingly confirming so in testing, I can't fathom how the functions are expected to work?

[andrewlecuyer]

Hi @B1ue-W01f! Are you still running into this issue and/or error?

Typically this sort of thing is related to a misconfiguration, e.g. a rogue repo configuration setting it causing pgBackRest to think it needs to create/configure an additional repo, with /var/lib/pgbackrest being pgBackRest's default repo path. CPK itself always mounts pgBackRest repos under /pgbackrest. So I would check and scrub your pgBackRest configs for any repo configs that should no longer be present.

Considering the age of this issue, I am going to proceed with closing. However, if you are still having trouble, please feel free to update this issue. Or you can also feel free to reach out via the PGO project community discord server