LINBIT
LINSTOR creates volumes larger than requested size
I create PersistentVolumeClaim with size of 1Gi:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: block-pvc
spec:
accessModes:
- ReadWriteMany
volumeMode: Block
resources:
requests:
storage: 1Gi
1Gi*1024*1024*1024 = 1073741824 bytes
But inside the pod the size of volume is only 1073750016:
/ # lsblk -bno SIZE /dev/drbd1014
1073750016
/ # qemu-img info /dev/drbd1014
image: /dev/drbd1014
file format: raw
virtual size: 1 GiB (1073750016 bytes)
disk size: 0 B
Child node '/file':
filename: /dev/drbd1014
protocol type: host_device
file length: 1 GiB (1073750016 bytes)
disk size: 0 B
link https://github.com/kubevirt/containerized-data-importer/issues/3159
In LINSTOR the volumes have different
"allocated_size_kib": 20976040,
"usable_size_kib": 20971528
Is it possible to order volumes in LINSTOR using usable_size, not allocated_size?
I just tried to use --gross but created volume even smaller than without this flag
Is there ant reliable way to request LINSTOR create a DRBD volume with exact size 1Gi (1073741824 bytes) that can be 100% usable by the client application?
Correct me if I'm wrong, but reading the linked issue it does not look like the question / request here is that LINSTOR should not give you a little bit more space, but exactly the space you asked for, right?
If so, please compare the sizes from linstor -m vd l | grep size_kib and linstor -m v l | grep '"allocated_size_kib":'. The volume-definition's size is exactly the size the user asked for, whereas the allocated_size_kib as well as the usable_size_kib might be a bit larger than the user wanted due to LVM / ZFS rounding up to the next extent size.
Getting back to your example, if you ask LINSTOR for a 1GiB large volume (i.e. 1073741824 bytes), you will get a bit more than 1GiB volume, in your case 1073750016. 1073750016 - 1073741824 = 8192, which means you got 8KiB more than requested.
Am I missing something or is the title of this issue the wrong way around?
Hi @ghernadi thank you for investigating, but seems I was wrong here, here is my pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prime-dbabbeef-a3f4-412e-aec6-cdbc8acd2c05
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: "1073741824"
storageClassName: replicated
volumeMode: Block
volumeName: pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
# linstor vd l -r pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
+-------------------------------------------------------------------------------------------+
| ResourceName | VolumeNr | VolumeMinor | Size | Gross | State |
|===========================================================================================|
| pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | 0 | 1000 | 1 GiB | | ok |
+-------------------------------------------------------------------------------------------+
# linstor v l -r pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
+---------------------------------------------------------------------------------------------------------------------------------+
| Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State |
|=================================================================================================================================|
| srv1 | pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | data | 0 | 1000 | /dev/drbd1000 | 1.00 GiB | Unused | UpToDate |
| srv2 | pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | data | 0 | 1000 | /dev/drbd1000 | 1.00 GiB | InUse | UpToDate |
| srv3 | pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | data | 0 | 1000 | /dev/drbd1000 | 1.00 GiB | Unused | UpToDate |
+---------------------------------------------------------------------------------------------------------------------------------+
# linstor -m v l -r pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
[
[
{
"name": "pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a",
"node_name": "srv1",
"props": {
"StorPoolName": "data"
},
"layer_object": {
"children": [
{
"type": "STORAGE",
"storage": {
"storage_volumes": [
{
"volume_number": 0,
"device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048848,
"usable_size_kib": 1048848,
"disk_state": "[]"
}
]
}
}
],
"type": "DRBD",
"drbd": {
"drbd_resource_definition": {
"peer_slots": 7,
"al_stripes": 1,
"al_stripe_size_kib": 32,
"port": 7001,
"transport_type": "IP",
"secret": "uV4aAOXbn/YtSIryS9yf",
"down": false
},
"node_id": 1,
"peer_slots": 7,
"al_stripes": 1,
"al_size": 32,
"flags": [
"INITIALIZED"
],
"drbd_volumes": [
{
"drbd_volume_definition": {
"volume_number": 0,
"minor_number": 1000
},
"device_path": "/dev/drbd1000",
"backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048840,
"usable_size_kib": 1048584
}
],
"connections": {
"srv3": {
"connected": true,
"message": "Connected"
},
"srv2": {
"connected": true,
"message": "Connected"
}
},
"promotion_score": 10103,
"may_promote": false
}
},
"state": {
"in_use": false
},
"uuid": "98460b7d-97cb-44d7-8022-2970006aa361",
"create_timestamp": 1728646426440,
"volumes": [
{
"volume_number": 0,
"storage_pool_name": "data",
"provider_kind": "ZFS",
"device_path": "/dev/drbd1000",
"allocated_size_kib": 1048848,
"state": {
"disk_state": "UpToDate"
},
"layer_data_list": [
{
"type": "DRBD",
"data": {
"drbd_volume_definition": {
"volume_number": 0,
"minor_number": 1000
},
"device_path": "/dev/drbd1000",
"backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048840,
"usable_size_kib": 1048584
}
},
{
"type": "STORAGE",
"data": {
"volume_number": 0,
"device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048848,
"usable_size_kib": 1048848,
"disk_state": "[]"
}
}
],
"uuid": "293063cd-fdd3-4e45-9178-802c61832290"
}
]
},
{
"name": "pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a",
"node_name": "srv2",
"props": {
"StorPoolName": "data"
},
"layer_object": {
"children": [
{
"type": "STORAGE",
"storage": {
"storage_volumes": [
{
"volume_number": 0,
"device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048848,
"usable_size_kib": 1048848,
"disk_state": "[]"
}
]
}
}
],
"type": "DRBD",
"drbd": {
"drbd_resource_definition": {
"peer_slots": 7,
"al_stripes": 1,
"al_stripe_size_kib": 32,
"port": 7001,
"transport_type": "IP",
"secret": "uV4aAOXbn/YtSIryS9yf",
"down": false
},
"node_id": 0,
"peer_slots": 7,
"al_stripes": 1,
"al_size": 32,
"flags": [
"INITIALIZED"
],
"drbd_volumes": [
{
"drbd_volume_definition": {
"volume_number": 0,
"minor_number": 1000
},
"device_path": "/dev/drbd1000",
"backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048840,
"usable_size_kib": 1048584
}
],
"connections": {
"srv3": {
"connected": true,
"message": "Connected"
},
"srv1": {
"connected": true,
"message": "Connected"
}
},
"promotion_score": 10103,
"may_promote": false
}
},
"state": {
"in_use": true
},
"uuid": "9cffb7f6-bea8-48a0-9906-d74e8eb2694c",
"create_timestamp": 1728646425721,
"volumes": [
{
"volume_number": 0,
"storage_pool_name": "data",
"provider_kind": "ZFS",
"device_path": "/dev/drbd1000",
"allocated_size_kib": 1048848,
"state": {
"disk_state": "UpToDate"
},
"layer_data_list": [
{
"type": "DRBD",
"data": {
"drbd_volume_definition": {
"volume_number": 0,
"minor_number": 1000
},
"device_path": "/dev/drbd1000",
"backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048840,
"usable_size_kib": 1048584
}
},
{
"type": "STORAGE",
"data": {
"volume_number": 0,
"device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048848,
"usable_size_kib": 1048848,
"disk_state": "[]"
}
}
],
"uuid": "66fbfb01-a4da-4643-8b50-dc33e97cf18d"
}
]
},
{
"name": "pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a",
"node_name": "srv3",
"props": {
"StorPoolName": "data"
},
"layer_object": {
"children": [
{
"type": "STORAGE",
"storage": {
"storage_volumes": [
{
"volume_number": 0,
"device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048848,
"usable_size_kib": 1048848,
"disk_state": "[]"
}
]
}
}
],
"type": "DRBD",
"drbd": {
"drbd_resource_definition": {
"peer_slots": 7,
"al_stripes": 1,
"al_stripe_size_kib": 32,
"port": 7001,
"transport_type": "IP",
"secret": "uV4aAOXbn/YtSIryS9yf",
"down": false
},
"node_id": 2,
"peer_slots": 7,
"al_stripes": 1,
"al_size": 32,
"flags": [
"INITIALIZED"
],
"drbd_volumes": [
{
"drbd_volume_definition": {
"volume_number": 0,
"minor_number": 1000
},
"device_path": "/dev/drbd1000",
"backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048840,
"usable_size_kib": 1048584
}
],
"connections": {
"srv1": {
"connected": true,
"message": "Connected"
},
"srv2": {
"connected": true,
"message": "Connected"
}
},
"promotion_score": 10103,
"may_promote": false
}
},
"state": {
"in_use": false
},
"uuid": "b9441c59-3f16-413e-b8b1-b73918d098f0",
"create_timestamp": 1728646426396,
"volumes": [
{
"volume_number": 0,
"storage_pool_name": "data",
"provider_kind": "ZFS",
"device_path": "/dev/drbd1000",
"allocated_size_kib": 1048848,
"state": {
"disk_state": "UpToDate"
},
"layer_data_list": [
{
"type": "DRBD",
"data": {
"drbd_volume_definition": {
"volume_number": 0,
"minor_number": 1000
},
"device_path": "/dev/drbd1000",
"backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048840,
"usable_size_kib": 1048584
}
},
{
"type": "STORAGE",
"data": {
"volume_number": 0,
"device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
"allocated_size_kib": 1048848,
"usable_size_kib": 1048848,
"disk_state": "[]"
}
}
],
"uuid": "7fa0181c-3522-4a44-826c-ea027b35558e"
}
]
}
]
]
For DRBD layer it is:
"allocated_size_kib": 1048840,
"usable_size_kib": 1048584
/ # lsblk -bno SIZE /dev/drbd1000
1073750016
>>> 1073750016/1024/1024/1024
1.0000076293945312
It seems CDI issue, I will continue investigating, and let you know. /close for now
It seems there is a bug (or feature?) in CDI. It expects that volume will have exact size that requested in PVC.
Personally I don't see the problem if the drive will be a n bit larger than expected, but it might affect some mechanics of copying PVCs between each other.
I didn't dig so far, but I see a potential issue, if CDI creates a new PVC with the same size as origin (and it reads this size from the PVC spec), but in fact that origin have larger volume inside it might block copying operations.
I think LINSTOR should create drives with exact requested size from PVC, if it is not possible due to technical limitations, it should at least report correct usable_size into created PV after the volume is created
The easiest way to get exact sizes is to use external meta data for DRBD and to request a volume size that is aligned with the allocation granularity of the backing storage used by the storage pool that the LINSTOR volume is created in. There is also an option that could be written into DRBD configuration files to force a specific size (equal to or smaller than the actual volume size), but LINSTOR does not normally set this option, because it complicates resizing volumes in various ways.
