dynamic-localpv-provisioner
dynamic-localpv-provisioner copied to clipboard
Published
20 hours ago •
openebs
openebs
xfs quota: wrong soft/hard limits number was set by openebs provisioner
Describe the bug: A clear and concise description of what the bug is. I installed openebs localpv provisioner via helm chart with xfs quota enabled:
xfsQuota:
# If true, enables XFS project quota
enabled: true
# Detailed configuration options for XFS project quota.
# If XFS Quota is enabled with the default values, the usage limit
# is set at the storage capacity specified in the PVC.
softLimitGrace: "60%"
hardLimitGrace: "90%"
Then I created a pvc with 10Gi and running it with a busybox container. However I found openebs set a wrong&bigger soft&hard limits for this pvc.
root@stonetest:~# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
busybox-test Bound pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2 10Gi RWO openebs-hostpath 4h51m
root@stonetest:~# kubectl get pv pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2 -o yaml | grep path
openebs.io/cas-type: local-hostpath
path: /openebs/local/pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2
storageClassName: openebs-hostpath
root@stonetest:~# mount | grep 45a4
/dev/vdc on /var/lib/kubelet/pods/6f140003-770a-408c-a281-3b1e1faecf0c/volumes/kubernetes.io~local-volume/pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2 type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
root@stonetest:~# lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
vdc xfs f993bbb1-d875-4436-ab4d-d7275b2c719c 30.6G 39% /var/lib/kubelet/pods/93ae9893-0374-4197-9084-02f64d8aaba6/volumes/kubernetes.io~local-volume/pvc-e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c
/var/lib/kubelet/pods/6f140003-770a-408c-a281-3b1e1faecf0c/volumes/kubernetes.io~local-volume/pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2
/openebs
root@stonetest:~# xfs_quota -x
xfs_quota> print
Filesystem Pathname
/openebs /dev/vdc (pquota)
/var/lib/kubelet/pods/6f140003-770a-408c-a281-3b1e1faecf0c/volumes/kubernetes.io~local-volume/pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2 /dev/vdc (pquota)
/var/lib/kubelet/pods/93ae9893-0374-4197-9084-02f64d8aaba6/volumes/kubernetes.io~local-volume/pvc-e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c /dev/vdc (pquota)
xfs_quota> report
Project quota on /openebs (/dev/vdc)
Blocks
Project ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
#0 0 0 0 00 [0 days]
#1 0 17179872 20401096 00 [--------]
#2 0 17179872 20401096 00 [--------]
The openebs provisioner set the soft limit as 17179872KB and hard limit as 20401096KB which exceeded the pvc's capacity (10Gi), I think this is wrong.
The soft limit should be 10Gi * 0.6 and the hard limit should be 10Gi * 0.9 respectively.
Expected behaviour: A concise description of what you expected to happen The openebs provisioner set the correct soft&hard limits for the pvc
Steps to reproduce the bug:
- attach a disk to k8s node, format it with xfs and mount it to /openebs with pquota options.
- install openebs localpv provisioner with helm chart v3.3.1.
root@stonetest:~# helm install openebs openebs/openebs --namespace openebs -f openebs-values.yaml
root@stonetest:~# cat openebs-values.yaml
apiserver:
enabled: false
varDirectoryPath:
baseDir: "/openebs"
provisioner:
enabled: false
localprovisioner:
enabled: true
basePath: "/openebs/local"
deviceClass:
enabled: false
hostpathClass:
# Name of the default hostpath StorageClass
name: openebs-hostpath
# If true, enables creation of the openebs-hostpath StorageClass
enabled: true
# Available reclaim policies: Delete/Retain, defaults: Delete.
reclaimPolicy: Delete
# If true, sets the openebs-hostpath StorageClass as the default StorageClass
isDefaultClass: false
# Path on the host where local volumes of this storage class are mounted under.
# NOTE: If not specified, this defaults to the value of localprovisioner.basePath.
basePath: "/openebs/local"
# Custom node affinity label(s) for example "openebs.io/node-affinity-value"
# that will be used instead of hostnames
# This helps in cases where the hostname changes when the node is removed and
# added back with the disks still intact.
# Example:
# nodeAffinityLabels:
# - "openebs.io/node-affinity-key-1"
# - "openebs.io/node-affinity-key-2"
nodeAffinityLabels: []
# Prerequisite: XFS Quota requires an XFS filesystem mounted with
# the 'pquota' or 'prjquota' mount option.
xfsQuota:
# If true, enables XFS project quota
enabled: true
# Detailed configuration options for XFS project quota.
# If XFS Quota is enabled with the default values, the usage limit
# is set at the storage capacity specified in the PVC.
softLimitGrace: "60%"
hardLimitGrace: "90%"
# Prerequisite: EXT4 Quota requires an EXT4 filesystem mounted with
# the 'prjquota' mount option.
ext4Quota:
# If true, enables XFS project quota
enabled: false
# Detailed configuration options for EXT4 project quota.
# If EXT4 Quota is enabled with the default values, the usage limit
# is set at the storage capacity specified in the PVC.
softLimitGrace: "0%"
hardLimitGrace: "0%"
snapshotOperator:
enabled: false
ndm:
enabled: false
ndmOperator:
enabled: false
ndmExporter:
enabled: false
webhook:
enabled: false
crd:
enableInstall: false
policies:
monitoring:
enabled: false
analytics:
enabled: false
jiva:
enabled: false
openebsLocalpv:
enabled: false
localpv-provisioner:
openebsNDM:
enabled: false
cstor:
enabled: false
openebsNDM:
enabled: false
openebs-ndm:
enabled: false
localpv-provisioner:
enabled: false
openebsNDM:
enabled: false
zfs-localpv:
enabled: false
lvm-localpv:
enabled: false
nfs-provisioner:
enabled: false
- create pvc and running a busybox with it
- xfs_quota -x then check the soft/hard limits for the pvc
The output of the following commands will help us better understand what's going on:
kubectl get pods -n <openebs_namespace> --show-labels
root@stonetest:~# kubectl get pods -n openebs --show-labels
NAME READY STATUS RESTARTS AGE LABELS
openebs-localpv-provisioner-5757b495fc-4zflv 1/1 Running 0 5h16m app=openebs,component=localpv-provisioner,name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=3.3.0,pod-template-hash=5757b495fc,release=openebs
kubectl logs <upgrade_job_pod> -n <openebs_namespace>
root@stonetest:~# kubectl -n openebs logs openebs-localpv-provisioner-5757b495fc-4zflv
I1207 03:10:43.992737 1 start.go:66] Starting Provisioner...
I1207 03:10:44.018165 1 start.go:130] Leader election enabled for localpv-provisioner via leaderElectionKey
I1207 03:10:44.018641 1 leaderelection.go:248] attempting to acquire leader lease openebs/openebs.io-local...
I1207 03:10:44.027045 1 leaderelection.go:258] successfully acquired lease openebs/openebs.io-local
I1207 03:10:44.027209 1 controller.go:810] Starting provisioner controller openebs.io/local_openebs-localpv-provisioner-5757b495fc-4zflv_9670bed1-7b43-4c70-b187-7f4e514cd24e!
I1207 03:10:44.027181 1 event.go:282] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"openebs", Name:"openebs.io-local", UID:"4838eadf-1cf6-4cec-af4e-8edafba21e87", APIVersion:"v1", ResourceVersion:"2999260", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' openebs-localpv-provisioner-5757b495fc-4zflv_9670bed1-7b43-4c70-b187-7f4e514cd24e became leader
I1207 03:10:44.128323 1 controller.go:859] Started provisioner controller openebs.io/local_openebs-localpv-provisioner-5757b495fc-4zflv_9670bed1-7b43-4c70-b187-7f4e514cd24e!
I1207 03:13:30.749165 1 controller.go:1279] provision "default/busybox-test" class "openebs-hostpath": started
I1207 03:13:30.755533 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"busybox-test", UID:"45a4e4f7-8117-4725-a17b-e3446da4b7a2", APIVersion:"v1", ResourceVersion:"2999559", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/busybox-test"
I1207 03:13:30.757597 1 provisioner_hostpath.go:76] Creating volume pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2 at node with labels {map[kubernetes.io/hostname:stonetest]}, path:/openebs/local/pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2,ImagePullSecrets:[]
2022-12-07T03:13:45.869Z INFO app/provisioner_hostpath.go:130 {"eventcode": "local.pv.quota.success", "msg": "Successfully applied quota", "rname": "pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2", "storagetype": "hostpath"}
2022-12-07T03:13:45.869Z INFO app/provisioner_hostpath.go:214 {"eventcode": "local.pv.provision.success", "msg": "Successfully provisioned Local PV", "rname": "pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2", "storagetype": "hostpath"}
I1207 03:13:45.869571 1 controller.go:1384] provision "default/busybox-test" class "openebs-hostpath": volume "pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2" provisioned
I1207 03:13:45.869586 1 controller.go:1397] provision "default/busybox-test" class "openebs-hostpath": succeeded
I1207 03:13:45.869594 1 volume_store.go:212] Trying to save persistentvolume "pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2"
I1207 03:13:45.872951 1 volume_store.go:219] persistentvolume "pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2" saved
I1207 03:13:45.873060 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"busybox-test", UID:"45a4e4f7-8117-4725-a17b-e3446da4b7a2", APIVersion:"v1", ResourceVersion:"2999559", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-45a4e4f7-8117-4725-a17b-e3446da4b7a2
I1207 03:20:00.738275 1 controller.go:1279] provision "xfs/busybox-test" class "openebs-hostpath": started
I1207 03:20:00.746069 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"xfs", Name:"busybox-test", UID:"e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c", APIVersion:"v1", ResourceVersion:"3000306", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "xfs/busybox-test"
I1207 03:20:00.748003 1 provisioner_hostpath.go:76] Creating volume pvc-e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c at node with labels {map[kubernetes.io/hostname:stonetest]}, path:/openebs/local/pvc-e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c,ImagePullSecrets:[]
2022-12-07T03:20:08.822Z INFO app/provisioner_hostpath.go:130 {"eventcode": "local.pv.quota.success", "msg": "Successfully applied quota", "rname": "pvc-e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c", "storagetype": "hostpath"}
2022-12-07T03:20:08.822Z INFO app/provisioner_hostpath.go:214 {"eventcode": "local.pv.provision.success", "msg": "Successfully provisioned Local PV", "rname": "pvc-e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c", "storagetype": "hostpath"}
I1207 03:20:08.822262 1 controller.go:1384] provision "xfs/busybox-test" class "openebs-hostpath": volume "pvc-e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c" provisioned
I1207 03:20:08.822275 1 controller.go:1397] provision "xfs/busybox-test" class "openebs-hostpath": succeeded
I1207 03:20:08.822282 1 volume_store.go:212] Trying to save persistentvolume "pvc-e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c"
I1207 03:20:08.826234 1 volume_store.go:219] persistentvolume "pvc-e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c" saved
I1207 03:20:08.826654 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"xfs", Name:"busybox-test", UID:"e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c", APIVersion:"v1", ResourceVersion:"3000306", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-e31459f2-67e5-44fb-bd2f-c4b7f1bc9f9c
root@stonetest:~#
Anything else we need to know?: Add any other context about the problem here.
Environment details:
- OpenEBS version (use
kubectl get po -n openebs --show-labels): openebs helm chart v3.3.1 - Kubernetes version (use
kubectl version): v1.23.10 - Cloud provider or hardware configuration:
- OS (e.g:
cat /etc/os-release): Ubuntu 22.04 LTS - kernel (e.g:
uname -a): Linux stonetest 5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux - others:
Well seems this is working as "expected".
The limit is applied on top of the capacity. So if your PVC capacity is 10GiB and your limit is 60%, so the quote is set to 160%, and thus 16GiB. And btw the max limit is double the PVC capacity, and so 100%.
I'm not sure why it is done this way, perhaps because it doesn't make sense to set a size smaller than the capacity, but I do think it is very confusing and we should at least document this better.
Can we check if we need any changes to the existing documentation to clearly explain these parameters?
Documentation for XFS Quota is now on the website: https://openebs.io/docs/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/xfs-quota/enable-xfs-quota
Another place the documentation lives is on this repo: https://github.com/openebs/dynamic-localpv-provisioner/tree/develop/docs/tutorials/hostpath/xfs_quota
