unable to remap invalid link 'pg_wal'

[OmriDim]

We are having a major problem with one of our clusters, we lost the ability to restore it. We did have full s3 backup to use. We tried the pgo restore command a couple of time which caused us to lose the cluster data. At the moment 2 solutions failed for this specific cluster with the error "unable to remap invalid link 'pg_wal'": pgo restore cluster hippo --pitr-target '2021-09-26 10:00:00' --backup-opts='--type=time --repo-type=s3' --pgbackrest-storage-type=s3 --no-prompt -n postgres-operator and pgo create cluster hippostb --pgbackrest-storage-type 's3' --pgbackrest-s3-bucket *bucket-name* --pgbackrest-s3-endpoint 'on-prem-s3' --pgbackrest-s3-key '*key*' --pgbackrest-repo-path '/backrestrepo/hippo-backrest-shared-repo' --password-superuser '*password*' --password-replication '*password*' --pgbackrest-s3-key-secret '*secret-key*' --metrics -n postgres-operator --standby

We couldn't perform a pgo create --restore-from because the original cluster isn't functional. Both commands worked on a new clusters, but failed on old ones(old shared-repo-paths) We believe it has something to do with the fact that this cluster was created before we updated the operator from 4.5.1 to 4.6.1, as this error came up when we created standby clusters to more old clusters. We have done some pgbackrest digging and reproduced the pgbackrest restore command that failed, and reached the same problem, for some reason it cannot create the pg_wal symlink.

Please tell us about your environment: crunchy version 4.6.1 Where is this running ( Local, Cloud Provider): on prem openshift Container Image Tag: centos7-12.6 PostgreSQL Version: 12.6 Platform (Docker, Kubernetes, OpenShift): openshift Platform Version: openshift 4.5

thank you!

[eliranw]

+1

[eliranw]

We have this problem for a week now, impacting a production system. @jkatz can you give a hand?

[jkatz]

For immediate help / production support please see the support page.

Otherwise issues are reviewed as folks in the community can review them.

[prasadmadanayake]

Seems this is a issue with pgbackrest https://github.com/pgbackrest/pgbackrest/pull/1589 which fixed in 2.3.7 so this should be fixed with a backrest version upgrade

[benjaminjb]

Is this problem still on-going for you or fixed with a pgbackrest version update?

[andrewlecuyer]

Since there have not been any replies/updates on this one in quite some time, I am proceeding with closing this issue.

If you continue to run into any issues with restores, please feel free to submit a new issue.