feat: reapply pvc when application storage changes

[adisazhar123]

This is not ready for review. There is more work to be done. I have reopened it because the old PR was automatically closed as a result of closing the 3.6.12 branch.

Please see this old PR for historical context https://github.com/juju/juju/pull/20683.

In #20589, we created a storage constraint watcher. As part of this PR, the watcher is wired up in the app worker to update PVCs and reapply the statefulset.

This PR focuses on wiring up the worker and watcher to get the functionality running. Resilience improvements (e.g., idempotency, safeguarding scaling operations, etc.) will be addressed in a separate PR.

A high level workflow:

• a juju application-storage <app> <directive> command is issued to update an application's storage constraint
• the app worker using the watcher to listen for changes will be notified, in which the following operations will take place
  • delete the statefulset with cascade=orphan which removes the statefulset controller but keeps the pods running (and its associated PVCs)
  • create the PVCs with the changes in storage constraint (for e.g. it could be a change in storage size)
  • create a new statefulset object, copying the relevant fields from the old statefulset, and assign the new PVCs to the statefulset
  • reapply the statefulset
• now, subsequent scale ups will have new pods created with the new PVC

Checklist

• [x] Code style: imports ordered, good names, simple structure, etc
• [x] Comments saying why design decisions were made
• [x] Go unit tests, with comments saying what you're testing
• [ ] Integration tests, with comments saying what you're testing
• [ ] doc.go added or updated in changed packages

QA steps

Please make sure you have included the changes from #20719 e2e test this. You may do so by branching off to a new branch from the top of this and merge the changes from #20719. This ensures you have the juju application-storage command interface.

Bootstrap a controller.

juju bootstrap minikube mymini

Add a model

juju add-model myminimodel

Deploy postgresql

juju deploy postgresql-k8s --channel=14/stable --trust

Check that the PVC is 1GB

kubectl get pvc -n myminimodel

Output

NAME                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
postgresql-k8s-pgdata-08c2af3e-postgresql-k8s-0   Bound    pvc-62ae817c-471a-4538-9fe6-78b5cd4ca125   1Gi        RWO            standard       <unset>                 3m51s

Update the storage size to 2GB

juju application-storage postgresql-k8s pgdata=2G,kubernetes

Scale app from 1 to 2

juju scale-application postgresql-k8s 2

Output

postgresql-k8s scaled to 2 units

Wait and check new pods are up

kubectl get pods -n myminimodel

Output

NAME                             READY   STATUS    RESTARTS        AGE
modeloperator-7b577fd869-46rkh   1/1     Running   1 (9m26s ago)   23m
postgresql-k8s-0                 2/2     Running   0               6m13s
postgresql-k8s-1                 2/2     Running   0               67s

Check the new PVC is created with the correct 2GB storage size

kubectl get pvc -n myminimodel

Output

NAME                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
postgresql-k8s-pgdata-08c2af3e-postgresql-k8s-0   Bound    pvc-62ae817c-471a-4538-9fe6-78b5cd4ca125   1Gi        RWO            standard       <unset>                 13m
postgresql-k8s-pgdata-08c2af3e-postgresql-k8s-1   Bound    pvc-352f9b47-4daa-4a93-acd6-0eca56dbd360   2Gi        RWO            standard       <unset>                 8m23s

Documentation changes

N/A.

Links

Issue: Fixes #19829.

Jira card: JUJU-8445