ZFS_THIN resources always show full allocation

[alexzhc]

trafficstars

Version

linstor: v1.18.2 drbd: v9.1.7 zfs: 2.1.4 os: RHEL, Ubuntu

How to reproduce

$ linstor sp l -s ZfsThin
+-----------------------------------------------------------------------------------------------------------------------------+
| StoragePool | Node         | Driver   | PoolName         | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName |
|=============================================================================================================================|
| ZfsThin     | k8s-worker-3 | ZFS_THIN | linstor_thinpool |    96.41 GiB |     99.50 GiB | True         | Ok    |            |
| ZfsThin     | k8s-worker-4 | ZFS_THIN | linstor_thinpool |    96.41 GiB |     99.50 GiB | True         | Ok    |            |
| ZfsThin     | k8s-worker-5 | ZFS_THIN | linstor_thinpool |    96.41 GiB |     99.50 GiB | True         | Ok    |            |
+-----------------------------------------------------------------------------------------------------------------------------+

Create a 10G volume

$ linstor rd c testvol
$ linstor vd c testvol 10G
$ linstor r c testvol --auto-place 2 -s ZfsThin

It shows 10GiB Allocated

$ linstor v l -r testvol
+ kubectl exec -n piraeus-system -i deploy/piraeus-cs-controller -- linstor --no-utf8 v l -r testvol
+--------------------------------------------------------------------------------------------------------------------+
| Node         | Resource | StoragePool          | VolNr | MinorNr | DeviceName    | Allocated | InUse  |      State |
|====================================================================================================================|
| k8s-worker-3 | testvol  | ZfsThin              |     0 |    1002 | /dev/drbd1002 | 10.00 GiB | Unused |   UpToDate |
| k8s-worker-4 | testvol  | DfltDisklessStorPool |     0 |    1002 | /dev/drbd1002 |           | Unused | TieBreaker |
| k8s-worker-5 | testvol  | ZfsThin              |     0 |    1002 | /dev/drbd1002 | 10.00 GiB | Unused |   UpToDate |
+--------------------------------------------------------------------------------------------------------------------+

But zfs shows only 2MB used

# zfs list
NAME                             USED  AVAIL     REFER  MOUNTPOINT
linstor_thinpool                2.57M  96.4G       24K  none
linstor_thinpool/testvol_00000  2.24M  96.4G     2.24M  -

Suggestion

Print zfs USED property value as the value for resource Allocated

[ghernadi]

Unfortunately the answer is: "it is complicated" :)

# create two volumes, one thick and one thin, both same size
root@bravo:~# zfs create -V 1G scratch-zfs/thick1G
root@bravo:~# zfs create -V 1G -s scratch-zfs/thin1G

# create random 10M data in both volumes:
root@bravo:~# dd if=/dev/urandom of=/dev/scratch-zfs/thin1G bs=1M count=10 oflag=direct
root@bravo:~# dd if=/dev/urandom of=/dev/scratch-zfs/thick1G bs=1M count=10 oflag=direct

# create snapshots
root@bravo:~# zfs snapshot scratch-zfs/thin1G@snap1
root@bravo:~# zfs snapshot scratch-zfs/thick1G@snap1

# repeat, but this time with 20M 
root@bravo:~# dd if=/dev/urandom of=/dev/scratch-zfs/thin1G bs=1M count=20 oflag=direct
root@bravo:~# dd if=/dev/urandom of=/dev/scratch-zfs/thick1G bs=1M count=20 oflag=direct
root@bravo:~# zfs snapshot scratch-zfs/thin1G@snap2
root@bravo:~# zfs snapshot scratch-zfs/thick1G@snap2

# just for fun, once again random 30M data, but leave the new snapshots "empty"
root@bravo:~# dd if=/dev/urandom of=/dev/scratch-zfs/thin1G bs=1M count=30 oflag=direct
root@bravo:~# dd if=/dev/urandom of=/dev/scratch-zfs/thick1G bs=1M count=30 oflag=direct
root@bravo:~# zfs snapshot scratch-zfs/thin1G@snap3
root@bravo:~# zfs snapshot scratch-zfs/thick1G@snap3

# result:
root@bravo:~# zfs list -o name,refer,used,avail,volsize,type -t volume,snapshot
NAME                          REFER   USED  AVAIL  VOLSIZE  TYPE
scratch-zfs/thick1G           30.3M  1.09G  9.08G       1G  volume
scratch-zfs/thick1G@snap1     10.1M  10.1M      -       1G  snapshot
scratch-zfs/thick1G@snap2     20.2M  20.2M      -       1G  snapshot
scratch-zfs/thick1G@snap3     30.3M     0B      -       1G  snapshot
scratch-zfs/thin1G            30.3M  60.6M  8.05G       1G  volume
scratch-zfs/thin1G@snap1      10.1M  10.1M      -       1G  snapshot
scratch-zfs/thin1G@snap2      20.2M  20.2M      -       1G  snapshot
scratch-zfs/thin1G@snap3      30.3M     0B      -       1G  snapshot

root@bravo:~# zfs version
zfs-0.8.3-1ubuntu12.13
zfs-kmod-0.8.3-1ubuntu12.13

The example above shows that we cannot use the USED column, since it contains wrong values for Linstor for empty snapshots as well as for the volumes themselves (USED column for volumes also have the sum of USED columns of their snapshots)

Based on the count from the dd commands, the REFER column seems to work for us (which is also used by Linstor since v1.18.0-rc1)

Speaking of versions: can you show me in your version the output of zfs list -o name,refer,used,avail,volsize,type -t volume,snapshot with your zfs version? Maybe this issue is based on changes within ZFS....

[alexzhc]

$ zfs list -o name,refer,used,avail,volsize,type -t volume,snapshot
NAME                               REFER   USED  AVAIL  VOLSIZE  TYPE
linstor_thinpool/testvol_00000      231M   231M  96.2G    10.0G  volume

Version:

$ modinfo zfs
filename:       /lib/modules/4.15.0-188-generic/updates/dkms/zfs.ko
version:        2.1.4-0york0~18.04
license:        CDDL
author:         OpenZFS
description:    ZFS
alias:          devname:zfs
alias:          char-major-10-249
srcversion:     4C33AA1150C7B7C3013D2EC
depends:        spl,znvpair,icp,zlua,zzstd,zunicode,zcommon,zavl
retpoline:      Y
name:           zfs
vermagic:       4.15.0-188-generic SMP mod_unload modversions
signat:         PKCS#7
signer:
sig_key:
sig_hashalgo:   md4