cloudfoundry-incubator
After enlarge database persistence disk size, no change applied to the database pod
Is your feature request related to a problem? Please describe.
After kubecf deployed, try to enlarge the database persistence disk size with below change:
update values.yaml, change the database disk size from 20G to 40G
database:
instances: ~
persistence:
size: 20Gi
==>
database:
instances: ~
persistence:
size: 40Gi
then run helm upgrade, then pod database-0 terminated and start as a new one.
Then perform below check:
$k get qsts database -n kubecf -o yaml
...
volumeClaimTemplates:
- metadata:
name: pxc-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 40Gi
storageClassName: new-ibmc-vpc-block-10iops-tier
...
...
$k get sts database -n kubecf -o yaml
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: pxc-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: new-ibmc-vpc-block-10iops-tier
volumeMode: Filesystem
status:
phase: Pending
...
login the pod database-0 and check the disk size, no change applied to the disk, still 20G.
$ k exec -it database-0 sh -n kubecf
Defaulting container name to database.
Use 'kubectl describe pod/database-0 -n kubecf' to see all of the containers in this pod.
sh-4.4# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 99G 47G 48G 50% /
tmpfs 64M 0 64M 0% /dev
tmpfs 7.3G 0 7.3G 0% /sys/fs/cgroup
/dev/vda2 99G 47G 48G 50% /root
shm 64M 0 64M 0% /dev/shm
/dev/vdf 20G 430M 20G 3% /var/lib/mysql
tmpfs 7.3G 16K 7.3G 1% /etc/mysql/tls/certs
tmpfs 7.3G 16K 7.3G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 7.3G 0 7.3G 0% /proc/acpi
tmpfs 7.3G 0 7.3G 0% /proc/scsi
tmpfs 7.3G 0 7.3G 0% /sys/firmware
Describe the solution you'd like After enlarging the disk size and then applying the change with helm upgrade, there should be a smooth data migration from the original database disk to a new bigger disk and then have new disk mounted to the database pod.
We have created an issue in Pivotal Tracker to manage this:
https://www.pivotaltracker.com/story/show/174568809
The labels on this github issue will be updated when the story is started.
We discussed the issue and updated the story in trackers. Can you check if k8s can handle this automatically: https://github.com/kubernetes/kubernetes/issues/35941#issuecomment-509265125
Some storage classes do support this: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims
@manno Below is the storageclass, it doesn't have allowVolumeExpansion that is mentioned in https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims , is that the reason the persistent disk resizing failed?
root@shuck1:~# kubectl get storageclass ibmc-vpc-block-10iops-tier -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
armada-service: addon-vpc-block-csi-driver
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"armada-service":"addon-vpc-block-csi-driver","storageclass.beta.kubernetes.io/is-default-class":"true","version":"2.0.3_375"},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","app":"ibm-vpc-block-csi-driver","razee/force-apply":"true"},"name":"ibmc-vpc-block-10iops-tier"},"parameters":{"billingType":"hourly","classVersion":"1","csi.storage.k8s.io/fstype":"ext4","encrypted":"false","encryptionKey":"","profile":"10iops-tier","region":"","resourceGroup":"","sizeRange":"[10-2000]GiB","tags":"","zone":""},"provisioner":"vpc.block.csi.ibm.io","reclaimPolicy":"Delete"}
storageclass.beta.kubernetes.io/is-default-class: "true"
version: 2.0.3_375
creationTimestamp: "2020-08-26T02:56:08Z"
labels:
addonmanager.kubernetes.io/mode: Reconcile
app: ibm-vpc-block-csi-driver
razee/force-apply: "true"
name: ibmc-vpc-block-10iops-tier
resourceVersion: "13304267"
selfLink: /apis/storage.k8s.io/v1/storageclasses/ibmc-vpc-block-10iops-tier
uid: 967f5315-a65c-4a61-b436-2ac68c2fe085
parameters:
billingType: hourly
classVersion: "1"
csi.storage.k8s.io/fstype: ext4
encrypted: "false"
encryptionKey: ""
profile: 10iops-tier
region: ""
resourceGroup: ""
sizeRange: '[10-2000]GiB'
tags: ""
zone: ""
provisioner: vpc.block.csi.ibm.io
reclaimPolicy: Delete
volumeBindingMode: Immediate
After changing the size of persistent disk of database to 40G, the database statefulset still has 20G in volumeClaimTemplates as below, shouldn't the storage be 40G after the change?
$k get sts database -n kubecf -o yaml
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: pxc-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: new-ibmc-vpc-block-10iops-tier
volumeMode: Filesystem
status:
phase: Pending
Looking at the code I can see the quarks statefulset reconciler cannot update the spec.resources.requests fields directly.
Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘template’, and ‘updateStrategy’ are forbidden”
The process seems to be more complicated, than just updating the sts spec.
@JimmyMa
Our CI is not using a storage class that supports this. Can you check if you can manually change the size in the PVC, kubectl edit -n kubecf pvc pxc-data?
Using the local-path storageclass, I get error: persistentvolumeclaims "pvc-example-quarks-statefulset-0" could not be patched: persistentvolumeclaims "pvc-example-quarks-statefulset-0" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize.
Same error with 'ibmc-file-gold'.
I just edit the pvc, change it to request 40Gi.
$ k get pvc pxc-data-database-0 -n kubecf -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
volume.kubernetes.io/selected-node: 10.240.1.13
creationTimestamp: "2020-08-31T03:53:59Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
quarks.cloudfoundry.org/az-index: "0"
quarks.cloudfoundry.org/quarks-statefulset-name: database
name: pxc-data-database-0
namespace: kubecf
resourceVersion: "13728907"
selfLink: /api/v1/namespaces/kubecf/persistentvolumeclaims/pxc-data-database-0
uid: ba41790c-1537-4944-81d7-2f101b5d6333
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 40Gi
storageClassName: new-ibmc-vpc-block-10iops-tier
volumeMode: Filesystem
volumeName: pvc-ba41790c-1537-4944-81d7-2f101b5d6333
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 20Gi
phase: Bound
But after the above update, nothing changed.
$ k get pv -n kubecf
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-74b3326d-85d6-48e3-a0b1-efc38b303124 100Gi RWO Delete Bound kubecf/singleton-blobstore-pvc-singleton-blobstore-0 new-ibmc-vpc-block-10iops-tier 2d4h
pvc-7e04c4bc-3f51-4413-ab39-e618ed4ec4ea 40Gi RWO Delete Bound kubecf/diego-cell-ephemeral-diego-cell-0 new-ibmc-vpc-block-10iops-tier 2d4h
pvc-ba41790c-1537-4944-81d7-2f101b5d6333 20Gi RWO Delete Bound kubecf/pxc-data-database-0 new-ibmc-vpc-block-10iops-tier 2d4h
$ k get pvc -n kubecf
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
diego-cell-ephemeral-diego-cell-0 Bound pvc-7e04c4bc-3f51-4413-ab39-e618ed4ec4ea 40Gi RWO new-ibmc-vpc-block-10iops-tier 2d4h
pxc-data-database-0 Bound pvc-ba41790c-1537-4944-81d7-2f101b5d6333 20Gi RWO new-ibmc-vpc-block-10iops-tier 2d4h
singleton-blobstore-pvc-singleton-blobstore-0 Bound pvc-74b3326d-85d6-48e3-a0b1-efc38b303124 100Gi RWO new-ibmc-vpc-block-10iops-tier 2d4h
I guess k8s doesn't automatically resize, because the storageclass is missing the allowVolumeExpansion.
We resized an underlying volume manually in the past, then shelled (k9s, ...) into the container and ran resize2fs.