Support metadata on PVC claims

[QuingKhaos]

Overview

Support adding labels and annotations to the PVC claims, so PGO reconciles the wanted annotations and labels on the PVCs created.

Use Case

Starting with ODF 4.10, we can reclaim space on Ceph RDB by annotating the PVCs with the reclaimspace.csiaddons.openshift.io/schedule annotation to create a ReclaimSpaceCronJob for the PVC.

Enabling this behavior on specific PVCs requires to add the annotation. Creating a ReclaimSpaceCronJob doesn't work in a GitOps workflow, as the PGO PVC names have a dynamic suffix (except the repo host).

Desired Behavior

Feels weird to have metadata on the same level of the spec.

spec:
  instances:
    - name: db
      dataVolumeClaimSpec:
        metadata:
          labels:
          annotations:
        storageClassName: ocs-storagecluster-ceph-rbd
      walVolumeClaimSpec:
        metadata:
          labels:
          annotations:
        storageClassName: ocs-storagecluster-ceph-rbd
  backups:
    pgbackrest:
      repos:
        - name: repo1
          volume:
            volumeClaimSpec:
              metadata:
                labels:
                annotations:
              storageClassName: ocs-storagecluster-ceph-rbd

Another design possibility would be to introduce new template keys (like stafeful sets), while keeping the old ones for BC. Feels more naturally to have metadata and spec side-by-side.

spec:
  instances:
    - name: db
      dataVolumeClaimTemplate:
        metadata:
          labels:
          annotations:
        spec:
          storageClassName: ocs-storagecluster-ceph-rbd
      walVolumeClaimTemplate:
        metadata:
          labels:
          annotations:
        spec:
          storageClassName: ocs-storagecluster-ceph-rbd
  backups:
    pgbackrest:
      repos:
        - name: repo1
          volume:
            volumeClaimTemplate:
              metadata:
                labels:
                annotations:
              spec:
                storageClassName: ocs-storagecluster-ceph-rbd

Environment

• Platform: OpenShift
• Platform Version: 4.10
• PGO Operator Version: 5.2
• Storage: ODF 4.10

[dsessler7]

A story has been added to our dev backlog.

[waldner]

+1

[ti8m-dah]

Having the metadata templates exposed, for the generated instance PVCs, would be also incredibly helpful for situations where you have a backup tool like Velero, and you want to exclude the DB volumes from backup but not the rest of the volumes within the namespace. (i.e. when you utilise bg-backrest as your backup utility for the DB and don't need/want the redundancy)

Right now there is no GitOps way to really do this, as the necessary exclude label can only be applied manually (or with a post hook).