When docker overlay2.size is enabled, the actual size of the pvc request is always the overlay2.size setting size

[wutz]

Describe the bug: Always docker overlay2.size and not pvc request size.

Expected behaviour: The actual size should be the pvc request size.

Steps to reproduce the bug:

1. openebs localpv and docker in same xfs with prjquota enabled

2. enable overlay.size in docker daemon config

cat /etc/docker/daemon.json
{
 "storage-driver": "overlay2",
    "storage-opts": [
        "overlay2.size=10GB"
    ]
}

3. deploy a pod and pvc with size equal 5 GB

The output of the following commands will help us better understand what's going on:

• kubectl get pods -n <openebs_namespace> --show-labels

  NAME                                          READY   STATUS    RESTARTS   AGE   LABELS
  localpv-localpv-provisioner-7d59d8f6c-zdt5c   1/1     Running   0          22h   app=localpv-provisioner,chart=localpv-provisioner-4.0.0,component=localpv-provisioner,heritage=Helm,name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=4.0.0,pod-template-hash=7d59d8f6c,release=localpv
  

• kubectl logs <upgrade_job_pod> -n <openebs_namespace>

  I0330 07:08:37.197263       1 controller.go:1366] provision "default/localpv-vol" class "local-ssd": started
  I0330 07:08:37.201411       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"localpv-vol", UID:"115f2690-dc12-49a5-bd12-adb1b5824f52", APIVersion:"v1", ResourceVersion:"11761250", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/localpv-vol"
  I0330 07:08:37.203873       1 provisioner_hostpath.go:77] Creating volume pvc-115f2690-dc12-49a5-bd12-adb1b5824f52 at node with labels {map[kubernetes.io/hostname:cn002.zw1.local]}, path:/var/openebs/local/pvc-115f2690-dc12-49a5-bd12-adb1b5824f52,ImagePullSecrets:[]
  2024-03-30T07:08:45.254Z	INFO	app/provisioner_hostpath.go:131		{"eventcode": "local.pv.quota.success", "msg": "Successfully applied quota", "rname": "pvc-115f2690-dc12-49a5-bd12-adb1b5824f52", "storagetype": "hostpath"}
  2024-03-30T07:08:45.254Z	INFO	app/provisioner_hostpath.go:215		{"eventcode": "local.pv.provision.success", "msg": "Successfully provisioned Local PV", "rname": "pvc-115f2690-dc12-49a5-bd12-adb1b5824f52", "storagetype": "hostpath"}
  I0330 07:08:45.254731       1 controller.go:1449] provision "default/localpv-vol" class "local-ssd": volume "pvc-115f2690-dc12-49a5-bd12-adb1b5824f52" provisioned
  I0330 07:08:45.254744       1 controller.go:1462] provision "default/localpv-vol" class "local-ssd": succeeded
  I0330 07:08:45.254752       1 volume_store.go:212] Trying to save persistentvolume "pvc-115f2690-dc12-49a5-bd12-adb1b5824f52"
  I0330 07:08:45.257734       1 volume_store.go:219] persistentvolume "pvc-115f2690-dc12-49a5-bd12-adb1b5824f52" saved
  I0330 07:08:45.257856       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"localpv-vol", UID:"115f2690-dc12-49a5-bd12-adb1b5824f52", APIVersion:"v1", ResourceVersion:"11761250", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-115f2690-dc12-49a5-bd12-adb1b5824f52
  

Anything else we need to know?: Add any other context about the problem here.

Environment details:

• OpenEBS version (use kubectl get po -n openebs --show-labels): 4.0.0
• Kubernetes version (use kubectl version): v1.28.7+k3s1
• Cloud provider or hardware configuration: AMD EPYC 7452 32-Core Processor / Intel D3-S4510 Series 480GB TLC SATA 6Gbps
• OS (e.g: cat /etc/os-release): ubuntu 22.04
• kernel (e.g: uname -a): 5.15.0-97-generic
• others:

[wutz]

I notice that overwriting hostpathClass.name from openebs-hostpath to local-ssd will trigger this problem, as described in helm chart values. You must redeploy openebs without modifying hostpathClass.name to work properly

[wutz]

I should have followed the tutorial to create a new storageClass instead of renaming openebs-hostpath

[wutz]

The use of quota on the worker node is set correctly, but setting quota on the master node is still equivalent to docker overlay.size

The master and the worker nodes with different, the master node has taint node-role.kubernetes.io/control-plane:NoSchedule and docker overlay.size is 30GB

Create a size 50GB pvc, the localpv logs:

I0330 09:27:25.421180       1 controller.go:1366] provision "default/localpv-vol" class "openebs-hostpath": started
I0330 09:27:25.427411       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"localpv-vol", UID:"d6b4589f-1c60-449f-9a62-dba6849d09a7", APIVersion:"v1", ResourceVersion:"11873609", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/localpv-vol"
I0330 09:27:25.429837       1 provisioner_hostpath.go:77] Creating volume pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7 at node with labels {map[kubernetes.io/hostname:mn03.zw1.local]}, path:/var/openebs/local/pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7,ImagePullSecrets:[]
2024-03-30T09:27:31.477Z	INFO	app/provisioner_hostpath.go:131		{"eventcode": "local.pv.quota.success", "msg": "Successfully applied quota", "rname": "pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7", "storagetype": "hostpath"}
2024-03-30T09:27:31.478Z	INFO	app/provisioner_hostpath.go:215		{"eventcode": "local.pv.provision.success", "msg": "Successfully provisioned Local PV", "rname": "pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7", "storagetype": "hostpath"}
I0330 09:27:31.478134       1 controller.go:1449] provision "default/localpv-vol" class "openebs-hostpath": volume "pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7" provisioned
I0330 09:27:31.478150       1 controller.go:1462] provision "default/localpv-vol" class "openebs-hostpath": succeeded
I0330 09:27:31.478162       1 volume_store.go:212] Trying to save persistentvolume "pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7"
I0330 09:27:31.481748       1 volume_store.go:219] persistentvolume "pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7" saved
I0330 09:27:31.481862       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"localpv-vol", UID:"d6b4589f-1c60-449f-9a62-dba6849d09a7", APIVersion:"v1", ResourceVersion:"11873609", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-d6b4589f-1c60-449f-9a62-dba6849d09a7

[wutz]

https://github.com/openebs/dynamic-localpv-provisioner/blob/2f165df9e37cd60ec88772d20e0acd2b655a347c/cmd/provisioner-localpv/app/helper_hostpath.go#L292-L293

Generating a project id based on the above code may cause docker to compete for concurrency

[tiagolobocastro]

This could be somewhat related to https://github.com/openebs/dynamic-localpv-provisioner/issues/155 ?

[avishnu]

Investigation scoped for v4.3.