Unable to bootstrap cluster from PVC/Volume

[joryirving]

Overview

Hi team,

I'm trying to DR my postgres cluster. I currently backup to 3 repos:

1. NFS via PVC
2. R2
3. Minio

I've repeatedly and successfully restored postgres clusters on a new k8s cluster via minio, however I'm running into a scenario where trying to do that via PVC/Volume fails, as it doesn't create the pod/PVC postgres-repo-host-0 until after the cluster is up and running. If I remove and recreate the cluster it works fine, but from a "blank" install of the operator it fails.

Environment

Kubernetes 1.32.1 Bare metal install (Talos Linux) PGO ubi8-5..7.2-0 Postgres ubi-16.6-1 (16) Storage local-hostpath (openebs)

Steps to Reproduce

Install PGO operator from scratch. Create postgres cluster using dataSource.pgbackrest.repo.volume for the first time. Pod fails to find data to restore from. It appears to a condition where the PVC/Volume isn't created until after the cluster is successfully running.

EXPECTED

I'm able to successfully bootstrap a new cluster from a backup on an NFS system.

ACTUAL

The cluster hangs and is unable to bootstrap.

Logs

N/A, as I worked around it by bootstrapping from S3 to reduce downtime.

Additional Information

This is an example of my postgrescluster that tried (and failed) to restore from PVC https://github.com/joryirving/home-ops/blob/9614dc3d6bab8a53ddf7344890765e4f057c7827/kubernetes/main/apps/database/crunchy-postgres/cluster/cluster.yaml

Specifically here:

      repos:
        - name: repo1
          volume: &nfs
            volumeClaimSpec:
              storageClassName: nfs-slow #csi-driver-nfs
              volumeName: postgres-nfs
              accessModes: ["ReadWriteMany"]
              resources:
                requests:
                  storage: 1Mi
          schedules:
            full: "30 1 * * 0" # Sunday at 01:30
            differential: "30 1 * * 1-6" # Mon-Sat at 01:30
            incremental: "30 3-23 * * *" # Every hour except 01:30-2:30
        - name: repo2
          s3: &r2
            bucket: crunchy-pgo
            endpoint: ${R2_ENDPOINT}
            region: us-east-1 #https://developers.cloudflare.com/r2/api/s3/api/#bucket-region
          schedules:
            full: "30 2 * * 0" # Sunday at 02:30
            incremental: "30 2 * * 1-6/2" # Mon-Sat at 02:30, every 2nd day
        # - name: repo3
        #   s3: &minio
        #     bucket: postgresql
        #     endpoint: s3.jory.dev
        #     region: ca-west-1
        #   schedules:
        #     full: "15 1 * * 0" # Sunday at 01:15
        #     differential: "15 1 * * 1-6" # Mon-Sat at 01:15
        #     incremental: "15 3-23 * * *" # Every hour except 01:30-2:30
  dataSource:
    pgbackrest:
      stanza: db
      configuration: *backupConfig
      global: *backupFlag
      repo:
        name: repo1
        volume: *nfs
        # s3: *r2

I'm manually creating the PV for the PVC to bind to here: https://github.com/joryirving/home-ops/blob/9614dc3d6bab8a53ddf7344890765e4f057c7827/kubernetes/main/apps/database/crunchy-postgres/cluster/nfs-pvc.yaml

[andrewlecuyer]

Hi @joryirving, sorry you're having trouble with this.

One thing I did want to clarify is that spec.dataSource.pgbackrest is intended for use with cloud-based repos (e.g. repos using S3, GCS or Azure storage), rather than volume-based repos. This is likely why you're running into trouble (and is also the reason I just created https://github.com/CrunchyData/postgres-operator/pull/4083 to add some additional validation to this part of the spec). You'll also see this mentioned in the spec.dataSource.pgbackrest section of the CRD reference:

Defines a pgBackRest cloud-based data source that can be used to pre-populate the PostgreSQL data directory for a new PostgreSQL cluster using a pgBackRest restore.

Additionally, in order to help you out further, I was hoping to clarify one detail about your use-case. Are you looking to use the volume/NFS defined in repo1 to simply bootstrap the cluster (meaning you don't want the newly cloned cluster to use this repo for archiving and backups)? Or do you want the new cluster to use the volume/NFS defined for repo1 for archiving and backing-up the new cluster?

[joryirving]

I'm essentially using it in a home kubernetes cluster, and it's for DR/cluster bootstrapping.

Essentially I back up to 3 repos right now: Repo1 - PVC (NFS storage on my NAS) Repo2 - R2 (offsite/cloudflare) Repo3 - Minio (Docker on my NAS)

I would like to get rid of repo3, so I can drop minio as a dependency.

How it's currently working is that I backup hourly to repo1/3, with long-term retention, that way if I accidentally blow up my homelab, I can wipe the entire k8s cluster, and bootstrap where I left off with backups/WAL. It will essentially pick up where it left off as a bootstrap, and then continue backing up to that same location.

This works when I use Minio as a bootstrap location/backup location, but it fails to work since you can't bootstrap a cluster from a volume.

I'm not using it to clone the cluster elsewhere, just as a DR situation (which happens... often).

I'm curious what the use-case is for backing up to PVC, if you're unable to use it to pre-populate a DB, or am I missing something obvious there?

[Kajot-dev]

Resolution was found in: #3459 I think. IMO #4083 will make the operator inconsistent with the docs/crd reference which state that this can be done

[joryirving]

I'm doing this on a fresh kubernetes cluster, so there's no lingering PV.

Restores from PV/PVC work, bootstraps from PV/PVC do not.

[Kajot-dev]

I'm doing this on a fresh kubernetes cluster, so there's no lingering PV.

Restores from PV/PVC work, bootstraps from PV/PVC do not.

I tested this and you're right, the proposed solution in the issue I mentioned does not work

[Kariton]

Hi @joryirving, sorry you're having trouble with this.

One thing I did want to clarify is that spec.dataSource.pgbackrest is intended for use with cloud-based repos (e.g. repos using S3, GCS or Azure storage), rather than volume-based repos. This is likely why you're running into trouble (and is also the reason I just created #4083 to add some additional validation to this part of the spec). You'll also see this mentioned in the spec.dataSource.pgbackrest section of the CRD reference:

Defines a pgBackRest cloud-based data source that can be used to pre-populate the PostgreSQL data directory for a new PostgreSQL cluster using a pgBackRest restore.

Additionally, in order to help you out further, I was hoping to clarify one detail about your use-case. Are you looking to use the volume/NFS defined in repo1 to simply bootstrap the cluster (meaning you don't want the newly cloned cluster to use this repo for archiving and backups)? Or do you want the new cluster to use the volume/NFS defined for repo1 for archiving and backing-up the new cluster?

despite the info about cloud-based data source it is still listed as option here: https://access.crunchydata.com/documentation/postgres-operator/latest/references/crd/5.8.x/postgrescluster#postgresclusterspecdatasourcepgbackrestrepovolumevolumeclaimspec

PostgresCluster.spec.dataSource.pgbackrest.repo.volume.volumeClaimSpec Defines a PersistentVolumeClaim spec used to create and/or bind a volume

that said - i now run into the validation which returns:

PostgresCluster.postgres-operator.crunchydata.com "mend-renovate-ce" is invalid: spec.dataSource.pgbackrest: Invalid value: "object": Only S3, GCS or Azure repos can be used as a pgBackRest data source....

i simply tried the exact same thing as @joryirving

spec:
  backups:
    pgbackrest:
      configuration: &backupConfig
        - secret:
            name: ${APP}-crunchy-postgres
      global: &backupFlag
        # Global
        compress-type: bz2
        compress-level: "9"
        # NFS PVC
        repo1-block: y
        repo1-bundle: y
        repo1-path: /${NAMESPACE}/${APP}
        repo1-retention-full: "15"  # days
        repo1-retention-full-type: time
      jobs:
        ttlSecondsAfterFinished: 60
      manual:
        repoName: repo1
        options:
          - --type=full
      metadata:
        labels:
          app.kubernetes.io/name: ${APP}-crunchy-postgres-backup
      repos:
        - name: repo1
          volume: &repo1-volumeClaim
            volumeClaimSpec:
              volumeName: crunchydata-pgbackrest
              accessModes: ["ReadWriteOnce"]
              resources:
                requests:
                  storage: ${POSTGRES_BACKREST_CAPACITY:-10Gi}
          schedules:
            full: "15 6 * * 1"  # every Monday at 06:15
            differential: "15 6 * * 0,2-6"  # every day at 06:15 except Monday
            incremental: "15 1-5,7-23 * * *"  # every hour except 06:15
  dataSource:
    pgbackrest:
      stanza: db
      configuration: *backupConfig
      global: *backupFlag
      repo:
        name: repo1
        volume: *repo1-volumeClaim

i want to bootstrap the pgsql cluster from backup when it exists as DR plan.

volumeName references this PV:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: crunchydata-pgbackrest
  annotations:
    kustomize.toolkit.fluxcd.io/ssa: IfNotPresent
spec:
  capacity:
    storage: 1Mi
  accessModes: ["ReadWriteMany"]
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: ${BACKUP_TARGET_SERVER}
    path: ${BACKUP_TARGET_PATH}/crunchydata-pgbackrest

[DerFetzer]

@Kariton @joryirving I am facing the same problem. Did any of you find a solution?

[Kariton]

no. no real solution. now my DR strategy depends on a minio bucket...

[joryirving]

Same. I'm forced to run minio for the sole purpose of PSQL DR. I can't use R2/B2 as the WAL streaming gets me on the transaction costs.