Editing VMSingle storage size doesn't update the existing PVC

[cskinfill]

Editing the VMSingle storage size doesn't seem to change an existing PVC. Is that intentional? If so, in order to increase storage we would edit the PVC directly. If that was not the intention, bug?

[f41gh7]

Sry for delay, operator must update request size of PVC. Actual size of pvc must be changed by CSI driver. Storage class must support dynamic resizing, if not, you will see error at operator log.

Most cloud providers support such resizing, but sometimes it must be manually patched allowVolumeExpansion: true for storage class. https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims

[symmetriac]

Hi! Also ran into this issue, but with VMCluster. operator recreated vmstorage statefullset, the volumeClaimTemplates field was updated. But there were no changes in pvc. I had to manually edit each pvc. operator version v0.22.0 operator logs

{"level":"info","ts":1642776817.4079556,"logger":"factory","msg":"create or update vmstorage for cluster","controller":"vmstorage","cluster":"vmcluster-persistent"}
{"level":"info","ts":1642776817.4119234,"logger":"factory","msg":"vmstorage was found, updating it","controller":"vmstorage","cluster":"vmcluster-persistent"}
{"level":"info","ts":1642776817.4119503,"logger":"client_utils","msg":"must re-recreate sts, its pvc claim was changed","size-diff":"100Gi"}
{"level":"info","ts":1642776820.4852316,"logger":"client_utils","msg":"sts update is needed","sts":"vmstorage-vmcluster-persistent","currentVersion":"vmstorage-vmcluster-persistent-5cc5669f94","desiredVersion":"vmstorage-vmcluster-persistent-5cc5669f94"}
{"level":"info","ts":1642776820.4852595,"logger":"client_utils","msg":"checking if update needed","controller":"sts.rollingupdate","desiredVersion":"vmstorage-vmcluster-persistent-5cc5669f94","wasRecreated":true}
{"level":"info","ts":1642776820.5047886,"logger":"client_utils","msg":"starting pods update, checking updated, by not ready pods","controller":"sts.rollingupdate","desiredVersion":"vmstorage-vmcluster-persistent-5cc5669f94","wasRecreated":true,"updated pods count":0,"desired version":"vmstorage-vmcluster-persistent-5cc5669f94"}
{"level":"info","ts":1642776820.5048149,"logger":"client_utils","msg":"updating pod","controller":"sts.rollingupdate","desiredVersion":"vmstorage-vmcluster-persistent-5cc5669f94","wasRecreated":true,"pod":"vmstorage-vmcluster-persistent-0"}

[f41gh7]

Hi! Also ran into this issue, but with VMCluster. operator recreated vmstorage statefullset, the volumeClaimTemplates field was updated. But there were no changes in pvc. I had to manually edit each pvc. operator version v0.22.0 operator logs

{"level":"info","ts":1642776817.4079556,"logger":"factory","msg":"create or update vmstorage for cluster","controller":"vmstorage","cluster":"vmcluster-persistent"}
{"level":"info","ts":1642776817.4119234,"logger":"factory","msg":"vmstorage was found, updating it","controller":"vmstorage","cluster":"vmcluster-persistent"}
{"level":"info","ts":1642776817.4119503,"logger":"client_utils","msg":"must re-recreate sts, its pvc claim was changed","size-diff":"100Gi"}
{"level":"info","ts":1642776820.4852316,"logger":"client_utils","msg":"sts update is needed","sts":"vmstorage-vmcluster-persistent","currentVersion":"vmstorage-vmcluster-persistent-5cc5669f94","desiredVersion":"vmstorage-vmcluster-persistent-5cc5669f94"}
{"level":"info","ts":1642776820.4852595,"logger":"client_utils","msg":"checking if update needed","controller":"sts.rollingupdate","desiredVersion":"vmstorage-vmcluster-persistent-5cc5669f94","wasRecreated":true}
{"level":"info","ts":1642776820.5047886,"logger":"client_utils","msg":"starting pods update, checking updated, by not ready pods","controller":"sts.rollingupdate","desiredVersion":"vmstorage-vmcluster-persistent-5cc5669f94","wasRecreated":true,"updated pods count":0,"desired version":"vmstorage-vmcluster-persistent-5cc5669f94"}
{"level":"info","ts":1642776820.5048149,"logger":"client_utils","msg":"updating pod","controller":"sts.rollingupdate","desiredVersion":"vmstorage-vmcluster-persistent-5cc5669f94","wasRecreated":true,"pod":"vmstorage-vmcluster-persistent-0"}

Can you check, that storage class supports expansion:

kubectl describe storageclass class-name - field allowVolumeExpansion: true must present.

Other possible issue - statefulset and pvcs has different label values. It prevents pvcs list by operator. But it's strange. Probably, selector labels must be used for this case.

And pvc resizing must be executed after rolling update of statefulset.

[symmetriac]

Yes, expansion is enabled, otherwise the manual edition of pvc object wouldn't work. Statefulset and pvc has the same labels:

labels:
    app.kubernetes.io/component: monitoring
    app.kubernetes.io/instance: vmcluster-persistent
    app.kubernetes.io/name: vmstorage
    managed-by: vm-operator

[f41gh7]

Yes, expansion is enabled, otherwise the manual edition of pvc object wouldn't work. Statefulset and pvc has the same labels:

labels:
    app.kubernetes.io/component: monitoring
    app.kubernetes.io/instance: vmcluster-persistent
    app.kubernetes.io/name: vmstorage
    managed-by: vm-operator

Checked it and locally it works with my kind deployment. Note, actual status capacity size is not changed - only requests. It must be updated by storageclass provisioner, in my case it cannot resize volume:

kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: rancher.io/local-path
  name: vmstorage-db-vmstorage-example-vmcluster-persistent-0
spec:
  resources:
    requests:
      storage: 16Gi
status:
  capacity:
    storage: 10Gi
  phase: Bound

operator logged following message:

{"level":"info","ts":1643204611.002347,"logger":"client_utils","msg":"need to expand pvc","name":"vmstorage-db-vmstorage-example-vmcluster-persistent-0","size":"16Gi"}

If you have any additional details to reproduce your issue, can you share it?

[sdomme]

@f41gh7 Hi, we also have this issue with VMSingle on an EKS Cluster with EBS storage. Will try to give you some more information, but I'm afraid I don't have much more.

Kubernetes: v1.23 VictoriaMetrics operator: v0.25.1

StorageClass:

NAME               PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-sc (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   139d

relevant settings for VMSingle:

  spec:
    storage:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
    storageMetadata: {}

PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    ... 
  finalizers:
  - apps.victoriametrics.com/finalizer
  - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/component: monitoring
    app.kubernetes.io/instance: victoria-stack
    app.kubernetes.io/name: vmsingle
    managed-by: vm-operator
  name: vmsingle-victoria-stack
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: ebs-sc
  volumeMode: Filesystem
  volumeName: pvc-3669fa16-....
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  phase: Bound

operator logged following message after changing the storage request in VMSingle CR:

[victoria-stack-victoria-metrics-operator-6779f9d478-b6jdt] {"level":"info","ts":1663660410.7777543,"logger":"factory","msg":"reconciling pvc","vm.single.pvc.create":"victoria-stack"}
[victoria-stack-victoria-metrics-operator-6779f9d478-b6jdt] {"level":"info","ts":1663660410.7777836,"logger":"factory","msg":"volume requests isn't same, update required","vm.single.pvc.create":"victoria-stack"}
[victoria-stack-victoria-metrics-operator-6779f9d478-b6jdt] {"level":"info","ts":1663660411.294399,"logger":"controllers.VMSingle","msg":"vmsingle  reconciled","vmsingle":"victoria/victoria-stack"}

aws-ebs-csi-driver didn't log anything. The PVC remained exactly the same.

[karmicdude]

+1 from me. The same situation with Cinder CSI (Openstack) and EBS CSI. The VMSingle object changes, but the operator doesn't resize the pvc, you have to resize it manually.

[f41gh7]

Sorry for that, will take a look soon.

[f41gh7]

Should be fixed at the next release, current master branch contains actual fix.

[f41gh7]

Fixed at v0.29.1 release