kubernetes-csi-addons Update precedence for schedule

This patch updates the schedule parsing logic in the following manner:

A new configmap key is added: schedule-precedence. It can be one of pvc-first or sc-first.
pvc-first is the current implementation we have, that considers the schedule in order of PVC > NS > SC.
sc-first is the new DS specific flag that only considers SCs as source of truth for schedule.
The default if no configmap is present will be pvc-first i.e. the current implementation.

This change aims to put the control of managing RS/KR operations to the Storage Admins.

If an application has specific needs, the Admin can grant the necessary RBACs so that the app owner can modify the schedule on RS/KR CronJobs. One would achive it in the following manner.

Annotate the RS/KR CronJob with (keyrotation/reclaimspace).csiaddons.openshift.io/exclude=true
- The value of the annotation can be anything and is not read.
Edit the RS/KR CronJob and update the schedule field.

Once a CronJob has exclude set, the application owner is in control of the operations.

Note to reviewers: Please suggest better wordings for the used terms.

Sep 18 '24 14:09 black-dragon74

Info

It was decided to have the precedence like this so that it is easier for the admin to update the schedule on all the PVCs by just updating it on NS or SC. Without it the admin would need to update it on per PVC basis.

Since we do not have a controller that watches NS changes yet, updates to NS would not trigger a reconcile, but if a schedule is present on NS, it will be read and used while reconciling SC or PVC.

Testing

Using precedence: sc-first

❯ oc get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
rbd-pvc   Bound    pvc-9e86fef5-8882-4b0d-83ab-238633614272   1Gi        RWO            rook-ceph-block   <unset>                 3s

// Add annotation to the SC
❯ oc annotate sc/rook-ceph-block "keyrotation.csiaddons.openshift.io/schedule=*/20 * * * *" --overwrite
storageclass.storage.k8s.io/rook-ceph-block annotated

LOGS:
2024-10-08T13:02:59.979Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "4f050e1b-7907-4d66-93e9-cc764f8b2ff4", "SchedulePrecedence": "sc-first"}
2024-10-08T13:02:59.980Z        INFO    Adding annotation       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "4f050e1b-7907-4d66-93e9-cc764f8b2ff4", "KeyRotationSchedule": "*/20 * * * *", "Annotation": "{\"metadata\":{\"annotations\":{\"keyrotation.csiaddons.openshift.io/cronjob\":\"rbd-pvc-1728392579\",\"keyrotation.csiaddons.openshift.io/schedule\":\"*/20 * * * *\"}}}"}
2024-10-08T13:03:00.057Z        INFO    KubeAPIWarningLogger    unknown field "spec.jobTemplate.metadata.creationTimestamp"
2024-10-08T13:03:00.058Z        INFO    successfully created new encryptionkeyrotationcronjob   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "4f050e1b-7907-4d66-93e9-cc764f8b2ff4", "KeyRotationSchedule": "*/20 * * * *"}
2024-10-08T13:03:00.058Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "4f050e1b-7907-4d66-93e9-cc764f8b2ff4", "KeyRotationSchedule": "*/20 * * * *", "SchedulePrecedence": "sc-first"}
2024-10-08T13:03:00.058Z        INFO    Annotation not set, exiting reconcile   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "4f050e1b-7907-4d66-93e9-cc764f8b2ff4", "KeyRotationSchedule": "*/20 * * * *"}
2024-10-08T13:03:00.061Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "bcf874b2-c117-4a36-9ea5-c92dcb63214d", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "SchedulePrecedence": "sc-first"}
2024-10-08T13:03:00.098Z        INFO    no upcoming schedule, requeue with delay until next run {"controller": "encryptionkeyrotationcronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "EncryptionKeyRotationCronJob", "EncryptionKeyRotationCronJob": {"name":"rbd-pvc-1728392579","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc-1728392579", "reconcileID": "78e6b433-d4a4-4dcd-b577-05ff3b7593da", "now": "2024-10-08T13:03:00.098Z", "nextRun": "2024-10-08T13:20:00.000Z"}
2024-10-08T13:03:00.106Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "bcf874b2-c117-4a36-9ea5-c92dcb63214d", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *", "SchedulePrecedence": "sc-first"}
2024-10-08T13:03:00.106Z        INFO    Annotation not set, exiting reconcile   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "bcf874b2-c117-4a36-9ea5-c92dcb63214d", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *"}
2024-10-08T13:03:00.114Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "1bd0fa1e-d559-4b39-b7f1-3903085a1de6", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "SchedulePrecedence": "sc-first"}
2024-10-08T13:03:00.114Z        INFO    no upcoming schedule, requeue with delay until next run {"controller": "encryptionkeyrotationcronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "EncryptionKeyRotationCronJob", "EncryptionKeyRotationCronJob": {"name":"rbd-pvc-1728392579","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc-1728392579", "reconcileID": "0a91a6ea-29eb-461d-9f80-27d95ff55471", "now": "2024-10-08T13:03:00.114Z", "nextRun": "2024-10-08T13:20:00.000Z"}
2024-10-08T13:03:00.125Z        INFO    successfully updated encryptionkeyrotationcronjob       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "1bd0fa1e-d559-4b39-b7f1-3903085a1de6", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *"}
2024-10-08T13:03:00.125Z        INFO    Adding annotation       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "1bd0fa1e-d559-4b39-b7f1-3903085a1de6", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *", "Annotation": "{\"metadata\":{\"annotations\":{\"keyrotation.csiaddons.openshift.io/schedule\":\"*/20 * * * *\"}}}"}
2024-10-08T13:03:00.133Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "1bd0fa1e-d559-4b39-b7f1-3903085a1de6", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *", "SchedulePrecedence": "sc-first"}
2024-10-08T13:03:00.133Z        INFO    Annotation not set, exiting reconcile   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "1bd0fa1e-d559-4b39-b7f1-3903085a1de6", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *"}

❯ oc get encryptionkeyrotationcronjobs
NAME                 SCHEDULE       SUSPEND   ACTIVE   LASTSCHEDULE   AGE
rbd-pvc-1728392579   */20 * * * *                                     6s

// Update shcedule on PVC, it should be overwritten by the value of SC's annotation
❯ oc annotate pvc/rbd-pvc "keyrotation.csiaddons.openshift.io/schedule=*/15 * * * *" --overwrite
persistentvolumeclaim/rbd-pvc annotated

LOGS:
2024-10-08T13:03:54.045Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "9cc22796-743d-4c26-b7ce-d00ce87837d7", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "SchedulePrecedence": "sc-first"}
2024-10-08T13:03:54.076Z        INFO    successfully updated encryptionkeyrotationcronjob       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "9cc22796-743d-4c26-b7ce-d00ce87837d7", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *"}
2024-10-08T13:03:54.076Z        INFO    Adding annotation       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "9cc22796-743d-4c26-b7ce-d00ce87837d7", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *", "Annotation": "{\"metadata\":{\"annotations\":{\"keyrotation.csiaddons.openshift.io/schedule\":\"*/20 * * * *\"}}}"}
2024-10-08T13:03:54.094Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "9cc22796-743d-4c26-b7ce-d00ce87837d7", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *", "SchedulePrecedence": "sc-first"}
2024-10-08T13:03:54.094Z        INFO    Annotation not set, exiting reconcile   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "9cc22796-743d-4c26-b7ce-d00ce87837d7", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *"}
2024-10-08T13:03:54.096Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "d80eefba-858d-4b2c-ac81-051d157b7a0f", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "SchedulePrecedence": "sc-first"}
2024-10-08T13:03:54.109Z        INFO    successfully updated encryptionkeyrotationcronjob       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "d80eefba-858d-4b2c-ac81-051d157b7a0f", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *"}
2024-10-08T13:03:54.110Z        INFO    Adding annotation       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "d80eefba-858d-4b2c-ac81-051d157b7a0f", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *", "Annotation": "{\"metadata\":{\"annotations\":{\"keyrotation.csiaddons.openshift.io/schedule\":\"*/20 * * * *\"}}}"}
2024-10-08T13:03:54.120Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "d80eefba-858d-4b2c-ac81-051d157b7a0f", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *", "SchedulePrecedence": "sc-first"}
2024-10-08T13:03:54.120Z        INFO    Annotation not set, exiting reconcile   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "d80eefba-858d-4b2c-ac81-051d157b7a0f", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579", "KeyRotationSchedule": "*/20 * * * *"}

❯ oc get encryptionkeyrotationcronjobs
NAME                 SCHEDULE       SUSPEND   ACTIVE   LASTSCHEDULE   AGE
rbd-pvc-1728392579   */20 * * * *                                     55s       // */15 was not applied

// Mark the CronJob for exclusion
❯ oc annotate encryptionkeyrotationcronjob/rbd-pvc-1728392579 "keyrotation.csiaddons.openshift.io/exclude=true" --overwrite
encryptionkeyrotationcronjob.csiaddons.openshift.io/rbd-pvc-1728392579 annotated

// Annotate the SC, the new schedule should not reflect on CronJob
❯ oc annotate sc/rook-ceph-block "keyrotation.csiaddons.openshift.io/schedule=*/21 * * * *" --overwrite
storageclass.storage.k8s.io/rook-ceph-block annotated

LOGS:
2024-10-08T13:05:59.654Z        INFO    EncryptionKeyRotationCronJob is managed by the application owner, exiting reconcile     {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "39943113-5966-4cfc-a9ca-b24fa1e6b1d7", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728392579"}

// Edit the schedule manually, it should stay
❯ oc edit encryptionkeyrotationcronjob/rbd-pvc-1728392579
encryptionkeyrotationcronjob.csiaddons.openshift.io/rbd-pvc-1728392579 edited

❯ oc get encryptionkeyrotationcronjobs
NAME                 SCHEDULE       SUSPEND   ACTIVE   LASTSCHEDULE   AGE
rbd-pvc-1728392579   */22 * * * *                                     4m16s     // The schedule is not overwritten

// Remove the exclusion annotation
❯ oc annotate encryptionkeyrotationcronjob/rbd-pvc-1728392579 "keyrotation.csiaddons.openshift.io/exclude-" --overwrite
encryptionkeyrotationcronjob.csiaddons.openshift.io/rbd-pvc-1728392579 annotated

// Annotate the SC, the schedule should now reflect on the CronJob
❯ oc annotate sc/rook-ceph-block "keyrotation.csiaddons.openshift.io/schedule=*/25 * * * *" --overwrite
storageclass.storage.k8s.io/rook-ceph-block annotated

❯ oc get encryptionkeyrotationcronjobs
NAME                 SCHEDULE       SUSPEND   ACTIVE   LASTSCHEDULE   AGE
rbd-pvc-1728392579   */25 * * * *                                     5m58s

Using Precedence: pvc-first

// Annotate the SC
❯ oc annotate sc/rook-ceph-block "keyrotation.csiaddons.openshift.io/schedule=*/25 * * * *" --overwrite
storageclass.storage.k8s.io/rook-ceph-block annotated

❯ oc get encryptionkeyrotationcronjobs
NAME                 SCHEDULE       SUSPEND   ACTIVE   LASTSCHEDULE   AGE
rbd-pvc-1728394208   */25 * * * *                                     14s

// Logs
2024-10-08T13:30:08.778Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "fcde44ec-6b2e-48e9-be9c-2ff31236c9b5", "SchedulePrecedence": "pvc-first"}
2024-10-08T13:30:08.778Z        INFO    Adding annotation       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "fcde44ec-6b2e-48e9-be9c-2ff31236c9b5", "KeyRotationSchedule": "*/25 * * * *", "Annotation": "{\"metadata\":{\"annotations\":{\"keyrotation.csiaddons.openshift.io/cronjob\":\"rbd-pvc-1728394208\",\"keyrotation.csiaddons.openshift.io/schedule\":\"*/25 * * * *\"}}}"}
2024-10-08T13:30:08.806Z        INFO    successfully created new encryptionkeyrotationcronjob   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "fcde44ec-6b2e-48e9-be9c-2ff31236c9b5", "KeyRotationSchedule": "*/25 * * * *"}
2024-10-08T13:30:08.806Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "fcde44ec-6b2e-48e9-be9c-2ff31236c9b5", "KeyRotationSchedule": "*/25 * * * *", "SchedulePrecedence": "pvc-first"}
2024-10-08T13:30:08.806Z        INFO    Annotation not set, exiting reconcile   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "fcde44ec-6b2e-48e9-be9c-2ff31236c9b5", "KeyRotationSchedule": "*/25 * * * *"}
2024-10-08T13:30:08.809Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "809eb196-0033-40ad-aa43-fa79cb2ff176", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "SchedulePrecedence": "pvc-first"}
2024-10-08T13:30:08.829Z        INFO    no upcoming schedule, requeue with delay until next run {"controller": "encryptionkeyrotationcronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "EncryptionKeyRotationCronJob", "EncryptionKeyRotationCronJob": {"name":"rbd-pvc-1728394208","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc-1728394208", "reconcileID": "5748b8bf-61e3-4d45-bce8-ecbcc92303af", "now": "2024-10-08T13:30:08.829Z", "nextRun": "2024-10-08T13:50:00.000Z"}
2024-10-08T13:30:08.830Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "809eb196-0033-40ad-aa43-fa79cb2ff176", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/25 * * * *", "SchedulePrecedence": "pvc-first"}
2024-10-08T13:30:08.830Z        INFO    Annotation not set, exiting reconcile   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "809eb196-0033-40ad-aa43-fa79cb2ff176", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/25 * * * *"}
2024-10-08T13:30:08.830Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "b44fd3a0-7127-42ea-9915-332ae0e81e24", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "SchedulePrecedence": "pvc-first"}
2024-10-08T13:30:08.843Z        INFO    no upcoming schedule, requeue with delay until next run {"controller": "encryptionkeyrotationcronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "EncryptionKeyRotationCronJob", "EncryptionKeyRotationCronJob": {"name":"rbd-pvc-1728394208","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc-1728394208", "reconcileID": "56c72f8f-9241-4609-8416-b14a700fe527", "now": "2024-10-08T13:30:08.843Z", "nextRun": "2024-10-08T13:50:00.000Z"}
2024-10-08T13:30:08.848Z        INFO    successfully updated encryptionkeyrotationcronjob       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "b44fd3a0-7127-42ea-9915-332ae0e81e24", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/25 * * * *"}
2024-10-08T13:30:08.848Z        INFO    Adding annotation       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "b44fd3a0-7127-42ea-9915-332ae0e81e24", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/25 * * * *", "Annotation": "{\"metadata\":{\"annotations\":{\"keyrotation.csiaddons.openshift.io/schedule\":\"*/25 * * * *\"}}}"}
2024-10-08T13:30:08.855Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "b44fd3a0-7127-42ea-9915-332ae0e81e24", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/25 * * * *", "SchedulePrecedence": "pvc-first"}
2024-10-08T13:30:08.855Z        INFO    Annotation not set, exiting reconcile   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "b44fd3a0-7127-42ea-9915-332ae0e81e24", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/25 * * * *"}
2024-10-08T13:30:08.856Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "9cc3d55d-79af-4fc9-8f07-833a354dc805", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "SchedulePrecedence": "pvc-first"}
2024-10-08T13:30:08.863Z        INFO    successfully updated encryptionkeyrotationcronjob       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "9cc3d55d-79af-4fc9-8f07-833a354dc805", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/25 * * * *"}
2024-10-08T13:30:08.863Z        INFO    Adding annotation       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "9cc3d55d-79af-4fc9-8f07-833a354dc805", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/25 * * * *", "Annotation": "{\"metadata\":{\"annotations\":{\"keyrotation.csiaddons.openshift.io/schedule\":\"*/25 * * * *\"}}}"}
2024-10-08T13:30:08.871Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "9cc3d55d-79af-4fc9-8f07-833a354dc805", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/25 * * * *", "SchedulePrecedence": "pvc-first"}
2024-10-08T13:30:08.871Z        INFO    Annotation not set, exiting reconcile   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "9cc3d55d-79af-4fc9-8f07-833a354dc805", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/25 * * * *"}

// Annotate the PVC
❯ oc annotate pvc/rbd-pvc "keyrotation.csiaddons.openshift.io/schedule=*/26 * * * *" --overwrite
persistentvolumeclaim/rbd-pvc annotated

❯ oc get encryptionkeyrotationcronjobs
NAME                 SCHEDULE       SUSPEND   ACTIVE   LASTSCHEDULE   AGE
rbd-pvc-1728394208   */26 * * * *                                     44s

// Logs
2024-10-08T13:30:45.953Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "69f4b3cb-87bf-428c-9f63-4f2062996568", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "SchedulePrecedence": "pvc-first"}
2024-10-08T13:30:45.970Z        INFO    successfully updated encryptionkeyrotationcronjob       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "69f4b3cb-87bf-428c-9f63-4f2062996568", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/26 * * * *"}
2024-10-08T13:30:45.970Z        INFO    Adding annotation       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "69f4b3cb-87bf-428c-9f63-4f2062996568", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/26 * * * *", "Annotation": "{\"metadata\":{\"annotations\":{\"keyrotation.csiaddons.openshift.io/schedule\":\"*/26 * * * *\"}}}"}
2024-10-08T13:30:45.981Z        INFO    no upcoming schedule, requeue with delay until next run {"controller": "encryptionkeyrotationcronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "EncryptionKeyRotationCronJob", "EncryptionKeyRotationCronJob": {"name":"rbd-pvc-1728394208","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc-1728394208", "reconcileID": "a73844b1-25e1-4896-8467-c53a4427f2c2", "now": "2024-10-08T13:30:45.981Z", "nextRun": "2024-10-08T13:52:00.000Z"}


// Update schedule on SC, should have no effect on the PVC schedule
❯ oc annotate sc/rook-ceph-block "keyrotation.csiaddons.openshift.io/schedule=*/22 * * * *" --overwrite
storageclass.storage.k8s.io/rook-ceph-block annotated

❯ oc get encryptionkeyrotationcronjobs
NAME                 SCHEDULE       SUSPEND   ACTIVE   LASTSCHEDULE   AGE
rbd-pvc-1728394208   */26 * * * *                                     80s

// Logs
2024-10-08T13:31:26.773Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "a1e0d4c8-670b-413e-a145-d28f3528f0d2", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "SchedulePrecedence": "pvc-first"}
2024-10-08T13:31:26.786Z        INFO    successfully updated encryptionkeyrotationcronjob       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "a1e0d4c8-670b-413e-a145-d28f3528f0d2", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/26 * * * *"}
2024-10-08T13:31:26.786Z        INFO    Adding annotation       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "a1e0d4c8-670b-413e-a145-d28f3528f0d2", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/26 * * * *", "Annotation": "{\"metadata\":{\"annotations\":{\"keyrotation.csiaddons.openshift.io/schedule\":\"*/26 * * * *\"}}}"}


// Add the exclude annotation
❯ oc annotate encryptionkeyrotationcronjob/rbd-pvc-1728394208 "keyrotation.csiaddons.openshift.io/exclude=true" --overwrite
encryptionkeyrotationcronjob.csiaddons.openshift.io/rbd-pvc-1728394208 annotated

❯ oc get encryptionkeyrotationcronjobs
NAME                 SCHEDULE       SUSPEND   ACTIVE   LASTSCHEDULE   AGE
rbd-pvc-1728394208   */26 * * * *                                     2m7s

// Logs
2024-10-08T13:31:52.823Z        INFO    no upcoming schedule, requeue with delay until next run {"controller": "encryptionkeyrotationcronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "EncryptionKeyRotationCronJob", "EncryptionKeyRotationCronJob": {"name":"rbd-pvc-1728394208","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc-1728394208", "reconcileID": "33d9c552-fc61-4229-9f50-6c1f808869d5", "now": "2024-10-08T13:31:52.823Z", "nextRun": "2024-10-08T13:52:00.000Z"}


// Update the schedule manually on the CronJOb now
❯ oc edit encryptionkeyrotationcronjob/rbd-pvc-1728394208
encryptionkeyrotationcronjob.csiaddons.openshift.io/rbd-pvc-1728394208 edited

❯ oc get encryptionkeyrotationcronjobs
NAME                 SCHEDULE       SUSPEND   ACTIVE   LASTSCHEDULE   AGE
rbd-pvc-1728394208   */19 * * * *                                     2m33s

// Logs
2024-10-08T13:32:39.362Z        INFO    no upcoming schedule, requeue with delay until next run {"controller": "encryptionkeyrotationcronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "EncryptionKeyRotationCronJob", "EncryptionKeyRotationCronJob": {"name":"rbd-pvc-1728394208","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc-1728394208", "reconcileID": "8835ccfe-c7d3-4667-be51-bb34c79c37bb", "now": "2024-10-08T13:32:39.362Z", "nextRun": "2024-10-08T13:38:00.000Z"}


// Update the PVC schedule, should have no effect on cronjob schedule
❯ oc annotate pvc/rbd-pvc "keyrotation.csiaddons.openshift.io/schedule=*/18 * * * *" --overwrite
persistentvolumeclaim/rbd-pvc annotated

❯ oc get encryptionkeyrotationcronjobs
NAME                 SCHEDULE       SUSPEND   ACTIVE   LASTSCHEDULE   AGE
rbd-pvc-1728394208   */19 * * * *                                     3m18s

// Logs
2024-10-08T13:33:24.640Z        INFO    EncryptionKeyRotationCronJob is managed by the application owner, exiting reconcile     {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "7b154e75-04af-409e-8fdd-191d84faca83", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208"}

// Remove the exclude annotation
❯ oc annotate encryptionkeyrotationcronjob/rbd-pvc-1728394208 "keyrotation.csiaddons.openshift.io/exclude-" --overwrite
encryptionkeyrotationcronjob.csiaddons.openshift.io/rbd-pvc-1728394208 annotated

// Logs
2024-10-08T13:33:52.335Z        INFO    no upcoming schedule, requeue with delay until next run {"controller": "encryptionkeyrotationcronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "EncryptionKeyRotationCronJob", "EncryptionKeyRotationCronJob": {"name":"rbd-pvc-1728394208","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc-1728394208", "reconcileID": "83d6fecf-2c6d-4dd0-ad92-b22cb6d4fb78", "now": "2024-10-08T13:33:52.335Z", "nextRun": "2024-10-08T13:38:00.000Z"}


// Update the pvc scheudle now, it should update on cronjob as well
❯ oc annotate pvc/rbd-pvc "keyrotation.csiaddons.openshift.io/schedule=*/17 * * * *" --overwrite
persistentvolumeclaim/rbd-pvc annotated

❯ oc get encryptionkeyrotationcronjobs
NAME                 SCHEDULE       SUSPEND   ACTIVE   LASTSCHEDULE   AGE
rbd-pvc-1728394208   */17 * * * *                      19s            4m11s

// Logs
2024-10-08T13:34:17.378Z        INFO    Determining schedule using precedence   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "860ed599-e5d7-40e0-9a54-6639edcec020", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "SchedulePrecedence": "pvc-first"}
2024-10-08T13:34:17.395Z        INFO    successfully updated encryptionkeyrotationcronjob       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "860ed599-e5d7-40e0-9a54-6639edcec020", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/17 * * * *"}
2024-10-08T13:34:17.395Z        INFO    Adding annotation       {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc", "reconcileID": "860ed599-e5d7-40e0-9a54-6639edcec020", "EncryptionKeyrotationCronJobName": "rbd-pvc-1728394208", "KeyRotationSchedule": "*/17 * * * *", "Annotation": "{\"metadata\":{\"annotations\":{\"keyrotation.csiaddons.openshift.io/schedule\":\"*/17 * * * *\"}}}"}
2024-10-08T13:34:17.431Z        INFO    no upcoming schedule, requeue with delay until next run {"controller": "encryptionkeyrotationcronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "EncryptionKeyRotationCronJob", "EncryptionKeyRotationCronJob": {"name":"rbd-pvc-1728394208","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "rbd-pvc-1728394208", "reconcileID": "32764d25-3673-4e05-8c77-be29e0da6b13", "now": "2024-10-08T13:34:17.431Z", "nextRun": "2024-10-08T13:51:00.000Z"}

Regards

Sep 18 '24 14:09 black-dragon74

As we had a meeting about this, it would be good to include a summary of the discussion in this PR.

From what I remember, we want to prevent users (non admins) from interfering with space reclaim, which needs:

an option to disable using annotations on PVCs and Namespaces
an option for admins to allow users to create (or modify) the ReclaimSpaceJob/ReclaimSpaceCronJob in their namespace
backwards compatible, so an optional setting in the ConfigMap for the operator, default to current behavior

... did I forget something?

Oct 02 '24 16:10 nixpanic

@black-dragon74 the annotations should be mentioned in the documentation too. Please add a paragraph about those.

Oct 16 '24 08:10 nixpanic

@black-dragon74 the annotations should be mentioned in the documentation too. Please add a paragraph about those.

May I follow the documentation updates in a separate PR?

P.S: The upcoming disable operations are related to this PR and documentation would be similar as well.

Oct 16 '24 08:10 black-dragon74

May I follow the documentation updates in a separate PR?

My preference is to include a commit about it in this PR. There is a large chance it is forgotten otherwise.

Oct 17 '24 15:10 nixpanic