zfs
zfs copied to clipboard
openzfs
The ARC size (current) is ten times larger than the Max size (zfs_arc_max), it will result in "Out of memory: killed process".
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Proxmox VE (Debian GNU/Linux 12 (bookworm)) |
| Distribution Version | proxmox-ve 8.2.4 |
| Kernel Version | Linux erpband-pve1 6.8.8-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.8-2 (2024-06-24T09:00Z) x86_64 GNU/Linux |
| Architecture | amd64 |
| OpenZFS Version | zfs-2.2.4-pve1 zfs-kmod-2.2.4-pve1 |
Describe the problem you're observing
Immediately after importing the pool, the ARC size (current) starts growing and quickly exceeds the Max size (zfs_arc_max). It continues to grow until there is no free RAM left on the server. This causes an Out of memory: killed process. The server has a total of 64GB of RAM.
In arc_summary, I see that the Anonymous metadata size starts growing, and along with it, the ARC size (current) starts growing. When the Anonymous metadata size reaches 50.1 GiB and the Target size (adaptive) becomes 52.2 GiB, there is no free RAM left in the system.
arc_summary | grep -E 'ARC size (current)|Min size (hard limit)|Max size (high water)|Anonymous metadata size' ARC size (current): 2616.7 % 52.3 GiB Min size (hard limit): 50.0 % 1024.0 MiB Max size (high water): 2:1 2.0 GiB Anonymous metadata size: 99.5 % 50.9 GiB
Why is the ARC Max size limit (/sys/module/zfs/parameters/zfs_arc_max) set to 2GB being ignored? How is this possible?
What is Anonymous metadata size? I didn't find any information.
Describe how to reproduce the problem
A complete video demonstrating the problem. https://youtu.be/ygukivJmaKw?si=jF5vZMDQm7FyfUmw&t=436
I import the pool.
root@erpband-pve1:~# zpool import -d /dev/disk/by-partlabel/ zp-erp1-hdd
After a 3-5 minutes, the ARC size starts to grow.
arc_summary | grep -E 'ARC size \(current\)|Min size \(hard limit\)|Max size \(high water\)|Anonymous metadata size'
ARC size (current): 2616.7 % 52.3 GiB
Min size (hard limit): 50.0 % 1024.0 MiB
Max size (high water): 2:1 2.0 GiB
Anonymous metadata size: 99.5 % 50.9 GiB
The maximum ARC size is set to 2GB, but it is being ignored.
ARC min 1Gb
root@erpband-pve1:~# cat /sys/module/zfs/parameters/zfs_arc_min
1073741823
ARC max 2Gb
root@erpband-pve1:~# cat /sys/module/zfs/parameters/zfs_arc_max
2147483647
I wrote a script arc_monitor.txt that outputs the parameters ARC size (current), Anonymous metadata size, Max size (high water), and Min size (hard limit) in a table. It clearly shows how the Anonymous metadata size grows, and along with it, the ARC size (current) grows.
time ARC size (current) Anonymous metadata size Max size (high water) Min size (hard limit)
14:50:22 2.0 GiB 813.8 MiB 2.0 GiB 1024.0 MiB
14:50:23 2.0 GiB 1.1 GiB 2.0 GiB 1024.0 MiB
14:50:24 2.3 GiB 1.3 GiB 2.0 GiB 1024.0 MiB
14:50:26 2.5 GiB 1.6 GiB 2.0 GiB 1024.0 MiB
14:50:27 2.8 GiB 1.8 GiB 2.0 GiB 1024.0 MiB
14:50:28 3.1 GiB 2.1 GiB 2.0 GiB 1024.0 MiB
14:50:30 3.3 GiB 2.4 GiB 2.0 GiB 1024.0 MiB
time ARC size (current) Anonymous metadata size Max size (high water) Min size (hard limit)
14:50:31 3.6 GiB 2.6 GiB 2.0 GiB 1024.0 MiB
14:50:32 3.8 GiB 2.9 GiB 2.0 GiB 1024.0 MiB
...
...
...
14:54:26 50.6 GiB 49.1 GiB 2.0 GiB 1024.0 MiB
14:54:27 50.8 GiB 49.3 GiB 2.0 GiB 1024.0 MiB
time ARC size (current) Anonymous metadata size Max size (high water) Min size (hard limit)
14:54:28 51.0 GiB 49.5 GiB 2.0 GiB 1024.0 MiB
14:54:29 51.2 GiB 49.7 GiB 2.0 GiB 1024.0 MiB
14:54:31 51.3 GiB 49.9 GiB 2.0 GiB 1024.0 MiB
14:54:32 51.5 GiB 50.1 GiB 2.0 GiB 1024.0 MiB
14:54:33 51.7 GiB 50.2 GiB 2.0 GiB 1024.0 MiB
14:54:34 51.8 GiB 50.4 GiB 2.0 GiB 1024.0 MiB
14:54:36 52.0 GiB 50.5 GiB 2.0 GiB 1024.0 MiB
time ARC size (current) Anonymous metadata size Max size (high water) Min size (hard limit)
14:54:37 52.2 GiB 50.7 GiB 2.0 GiB 1024.0 MiB
14:54:38 52.3 GiB 50.9 GiB 2.0 GiB 1024.0 MiB
At this point, the system froze due to lack of RAM.
Additional information
root@erpband-pve1:~# zpool status -v -t -D
pool: zp-erp1-hdd
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zp-erp1-hdd ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
p-zp-erp1-hdd-rz2-d1 ONLINE 0 0 0 (trim unsupported)
p-zp-erp1-hdd-rz2-d2 ONLINE 0 0 0 (trim unsupported)
p-zp-erp1-hdd-rz2-d3 ONLINE 0 0 0 (trim unsupported)
p-zp-erp1-hdd-rz2-d4 ONLINE 0 0 0 (trim unsupported)
p-zp-erp1-hdd-rz2-d5 ONLINE 0 0 0 (trim unsupported)
p-zp-erp1-hdd-rz2-d6 ONLINE 0 0 0 (trim unsupported)
special
mirror-1 ONLINE 0 0 0
p-zp-erp1-hdd-nvme-spec-mi-d1 ONLINE 0 0 0 (untrimmed)
p-zp-erp1-hdd-nvme-spec-mi-d2 ONLINE 0 0 0 (untrimmed)
logs
mirror-2 ONLINE 0 0 0
p-zp-erp1-hdd-nvme-log-mi-d1 ONLINE 0 0 0 (untrimmed)
p-zp-erp1-hdd-nvme-log-mi-d2 ONLINE 0 0 0 (untrimmed)
cache
p-zp-erp1-hdd-nvme-cache-d1 ONLINE 0 0 0 (untrimmed)
p-zp-erp1-hdd-nvme-cache-d2 ONLINE 0 0 0 (untrimmed)
errors: No known data errors
dedup: DDT entries 211037580, size 313B on disk, 173B in core
bucket allocated referenced
______ ______________________________ ______________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
1 189M 23.6T 15.5T 16.3T 189M 23.6T 15.5T 16.3T
2 10.0M 1.25T 928G 962G 21.9M 2.74T 1.96T 2.04T
4 1.75M 224G 125G 135G 8.76M 1.10T 618G 667G
8 688K 86.1G 40.2G 44.9G 6.73M 861G 400G 448G
16 46.6K 5.83G 3.36G 3.63G 978K 122G 71.0G 76.5G
32 3.96K 507M 259M 284M 165K 20.6G 10.7G 11.7G
64 624 78M 26.3M 31.5M 51.8K 6.48G 2.13G 2.56G
128 158 19.8M 6.51M 7.57M 27.2K 3.40G 1.11G 1.29G
256 199 24.9M 12M 12.8M 71.6K 8.95G 4.56G 4.83G
512 95 11.9M 6.52M 6.76M 77.1K 9.64G 5.61G 5.77G
1K 32 4M 2.51M 2.58M 34.7K 4.34G 2.60G 2.68G
2K 7 896K 56K 112K 17.9K 2.24G 143M 287M
4K 1 128K 8K 16.0K 5.39K 690M 43.1M 86.1M
8K 1 128K 8K 16.0K 11.7K 1.47G 93.9M 188M
64K 1 128K 8K 16.0K 70.1K 8.77G 561M 1.09G
Total 201M 25.2T 16.6T 17.4T 228M 28.5T 18.6T 19.5T
pool: zp-erp1-nvme
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zp-erp1-nvme ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme-eui.002538ba31b4e232-part4 ONLINE 0 0 0 (100% trimmed, completed at Sun Jul 7 00:24:42 2024)
nvme-Samsung_SSD_980_PRO_1TB_S5GXNU0WC31300E_1-part4 ONLINE 0 0 0 (100% trimmed, completed at Sun Jul 7 00:24:42 2024)
errors: No known data errors
dedup: no DDT entries
root@erpband-pve1:~# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zp-erp1-hdd 44.0T 26.3T 17.7T - - 0% 59% 1.12x ONLINE -
raidz2-0 43.7T 26.2T 17.5T - - 0% 60.0% - ONLINE
p-zp-erp1-hdd-rz2-d1 7.28T - - - - - - - ONLINE
p-zp-erp1-hdd-rz2-d2 7.28T - - - - - - - ONLINE
p-zp-erp1-hdd-rz2-d3 7.28T - - - - - - - ONLINE
p-zp-erp1-hdd-rz2-d4 7.28T - - - - - - - ONLINE
p-zp-erp1-hdd-rz2-d5 7.28T - - - - - - - ONLINE
p-zp-erp1-hdd-rz2-d6 7.28T - - - - - - - ONLINE
special - - - - - - - - -
mirror-1 348G 83.7G 264G - - 53% 24.0% - ONLINE
p-zp-erp1-hdd-nvme-spec-mi-d1 350G - - - - - - - ONLINE
p-zp-erp1-hdd-nvme-spec-mi-d2 350G - - - - - - - ONLINE
logs - - - - - - - - -
mirror-2 6.50G 0 6.50G - - 0% 0.00% - ONLINE
p-zp-erp1-hdd-nvme-log-mi-d1 7G - - - - - - - ONLINE
p-zp-erp1-hdd-nvme-log-mi-d2 7G - - - - - - - ONLINE
cache - - - - - - - - -
p-zp-erp1-hdd-nvme-cache-d1 404G 122G 282G - - 0% 30.2% - ONLINE
p-zp-erp1-hdd-nvme-cache-d2 404G 122G 282G - - 0% 30.3% - ONLINE
zp-erp1-nvme 99.5G 41.0G 58.5G - - 50% 41% 1.00x ONLINE -
mirror-0 99.5G 41.0G 58.5G - - 50% 41.2% - ONLINE
nvme-eui.002538ba31b4e232-part4 100G - - - - - - - ONLINE
nvme-Samsung_SSD_980_PRO_1TB_S5GXNU0WC31300E_1-part4 100G - - - - - - - ONLINE
root@erpband-pve1:~# lsblk -o NAME,PARTLABEL,SIZE,PATH,mountpoint
NAME PARTLABEL SIZE PATH MOUNTPOINT
sda 7.3T /dev/sda
└─sda1 p-zp-erp1-hdd-rz2-d1 7.3T /dev/sda1
sdb 7.3T /dev/sdb
└─sdb1 p-zp-erp1-hdd-rz2-d2 7.3T /dev/sdb1
sdc 7.3T /dev/sdc
└─sdc1 p-zp-erp1-hdd-rz2-d3 7.3T /dev/sdc1
sdd 7.3T /dev/sdd
└─sdd1 p-zp-erp1-hdd-rz2-d4 7.3T /dev/sdd1
sde 7.3T /dev/sde
└─sde1 p-zp-erp1-hdd-rz2-d5 7.3T /dev/sde1
sdf 7.3T /dev/sdf
└─sdf1 p-zp-erp1-hdd-rz2-d6 7.3T /dev/sdf1
zd0 1M /dev/zd0
zd16 80G /dev/zd16
├─zd16p1 EFI system partition 100M /dev/zd16p1
├─zd16p2 Microsoft reserved partition 16M /dev/zd16p2
└─zd16p3 Basic data partition 79.9G /dev/zd16p3
nvme0n1 931.5G /dev/nvme0n1
├─nvme0n1p1 BIOS boot partition 3M /dev/nvme0n1p1
├─nvme0n1p2 EFI system partition 512M /dev/nvme0n1p2
├─nvme0n1p3 Linux RAID 70G /dev/nvme0n1p3
│ └─md0 69.9G /dev/md0
│ ├─pve-root 50G /dev/mapper/pve-root /
│ └─pve-swap 16G /dev/mapper/pve-swap [SWAP]
├─nvme0n1p4 p-zp-erp1-nvme-mi-d1 100G /dev/nvme0n1p4
├─nvme0n1p5 p-zp-erp1-hdd-nvme-spec-mi-d1 350G /dev/nvme0n1p5
├─nvme0n1p6 p-zp-erp1-hdd-nvme-cache-d1 404G /dev/nvme0n1p6
└─nvme0n1p7 p-zp-erp1-hdd-nvme-log-mi-d1 7G /dev/nvme0n1p7
nvme1n1 931.5G /dev/nvme1n1
├─nvme1n1p1 BIOS boot partition 3M /dev/nvme1n1p1
├─nvme1n1p2 EFI system partition 512M /dev/nvme1n1p2
├─nvme1n1p3 Linux RAID 70G /dev/nvme1n1p3
│ └─md0 69.9G /dev/md0
│ ├─pve-root 50G /dev/mapper/pve-root /
│ └─pve-swap 16G /dev/mapper/pve-swap [SWAP]
├─nvme1n1p4 p-zp-erp1-nvme-mi-d2 100G /dev/nvme1n1p4
├─nvme1n1p5 p-zp-erp1-hdd-nvme-spec-mi-d2 350G /dev/nvme1n1p5
├─nvme1n1p6 p-zp-erp1-hdd-nvme-cache-d2 404G /dev/nvme1n1p6
└─nvme1n1p7 p-zp-erp1-hdd-nvme-log-mi-d2 7G /dev/nvme1n1p7
root@erpband-pve1:~# zfs version
zfs-2.2.4-pve1
zfs-kmod-2.2.4-pve1
root@erpband-pve1:~# uname -a
Linux erpband-pve1 6.8.8-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.8-2 (2024-06-24T09:00Z) x86_64 GNU/Linux
root@erpband-pve1:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Release: 12
Codename: bookworm
root@erpband-pve1:~# zpool get all zp-erp1-hdd
NAME PROPERTY VALUE SOURCE
zp-erp1-hdd size 44.0T -
zp-erp1-hdd capacity 59% -
zp-erp1-hdd altroot - default
zp-erp1-hdd health ONLINE -
zp-erp1-hdd guid 12804097997330970578 -
zp-erp1-hdd version - default
zp-erp1-hdd bootfs - default
zp-erp1-hdd delegation on default
zp-erp1-hdd autoreplace off default
zp-erp1-hdd cachefile - default
zp-erp1-hdd failmode wait default
zp-erp1-hdd listsnapshots off default
zp-erp1-hdd autoexpand off default
zp-erp1-hdd dedupratio 1.12x -
zp-erp1-hdd free 17.9T -
zp-erp1-hdd allocated 26.1T -
zp-erp1-hdd readonly off -
zp-erp1-hdd ashift 13 local
zp-erp1-hdd comment - default
zp-erp1-hdd expandsize - -
zp-erp1-hdd freeing 0 -
zp-erp1-hdd fragmentation 0% -
zp-erp1-hdd leaked 0 -
zp-erp1-hdd multihost off default
zp-erp1-hdd checkpoint - -
zp-erp1-hdd load_guid 14916355245013050689 -
zp-erp1-hdd autotrim off default
zp-erp1-hdd compatibility off default
zp-erp1-hdd bcloneused 0 -
zp-erp1-hdd bclonesaved 0 -
zp-erp1-hdd bcloneratio 1.00x -
zp-erp1-hdd feature@async_destroy enabled local
zp-erp1-hdd feature@empty_bpobj active local
zp-erp1-hdd feature@lz4_compress active local
zp-erp1-hdd feature@multi_vdev_crash_dump enabled local
zp-erp1-hdd feature@spacemap_histogram active local
zp-erp1-hdd feature@enabled_txg active local
zp-erp1-hdd feature@hole_birth active local
zp-erp1-hdd feature@extensible_dataset active local
zp-erp1-hdd feature@embedded_data active local
zp-erp1-hdd feature@bookmarks enabled local
zp-erp1-hdd feature@filesystem_limits enabled local
zp-erp1-hdd feature@large_blocks enabled local
zp-erp1-hdd feature@large_dnode enabled local
zp-erp1-hdd feature@sha512 enabled local
zp-erp1-hdd feature@skein enabled local
zp-erp1-hdd feature@edonr enabled local
zp-erp1-hdd feature@userobj_accounting active local
zp-erp1-hdd feature@encryption enabled local
zp-erp1-hdd feature@project_quota active local
zp-erp1-hdd feature@device_removal enabled local
zp-erp1-hdd feature@obsolete_counts enabled local
zp-erp1-hdd feature@zpool_checkpoint enabled local
zp-erp1-hdd feature@spacemap_v2 active local
zp-erp1-hdd feature@allocation_classes active local
zp-erp1-hdd feature@resilver_defer enabled local
zp-erp1-hdd feature@bookmark_v2 enabled local
zp-erp1-hdd feature@redaction_bookmarks enabled local
zp-erp1-hdd feature@redacted_datasets enabled local
zp-erp1-hdd feature@bookmark_written enabled local
zp-erp1-hdd feature@log_spacemap active local
zp-erp1-hdd feature@livelist enabled local
zp-erp1-hdd feature@device_rebuild enabled local
zp-erp1-hdd feature@zstd_compress active local
zp-erp1-hdd feature@draid enabled local
zp-erp1-hdd feature@zilsaxattr enabled local
zp-erp1-hdd feature@head_errlog active local
zp-erp1-hdd feature@blake3 enabled local
zp-erp1-hdd feature@block_cloning enabled local
zp-erp1-hdd feature@vdev_zaps_v2 active local
Include any warning/errors/backtraces from the system logs
There are no errors or warnings in the logs. The pool imports successfully, but it consumes all the RAM. Here is the log file for the period from the pool import to the system freeze: logs_2024-07-07_18-40-36.txt .
@robn Thank you for your help!
you wrote at https://github.com/openzfs/zfs/issues/16322
Could you capture cat /proc/spl/kmem/slab before the import, and at the time of the OOM, and post them?
I did it.
Full log /proc/slabinfo and /proc/meminfo memory_info_2024-07-08_17-03-53.txt
Important note! If I import the pool as read-only, everything works fine, and the RAM is not consumed.
zpool import -o readonly=on -d /dev/disk/by-partlabel/ zp-erp1-hdd
Before import pool zp-erp1-hdd
root@erpband-pve1:~# cat /proc/spl/kmem/slab
--------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency ---
name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max
spl_zlib_workspace_cache 0x00080 0 0 2145216 268104 0 0 0 0 0 0 0 0 0
kcf_context_cache 0x00100 - 0 - 40 - - - - 0 - - - -
zfs_btree_leaf_cache 0x00100 - 135168 - 4096 - - - - 33 - - - -
metaslab_alloc_trace_cache 0x00100 - 0 - 72 - - - - 0 - - - -
brt_entry_cache 0x00100 - 0 - 40 - - - - 0 - - - -
brt_pending_entry_cache 0x00100 - 0 - 160 - - - - 0 - - - -
ddt_cache 0x00080 995520 794880 199104 24840 5 4 4 40 32 32 0 0 0
ddt_entry_cache 0x00100 - 0 - 448 - - - - 0 - - - -
zio_cache 0x00100 - 46080 - 1280 - - - - 36 - - - -
zio_link_cache 0x00100 - 0 - 48 - - - - 0 - - - -
zio_buf_comb_512 0x00102 - 4096 - 512 - - - - 8 - - - -
zio_buf_comb_1024 0x00102 - 1024 - 1024 - - - - 1 - - - -
zio_buf_comb_1536 0x00102 - 3072 - 1536 - - - - 2 - - - -
zio_buf_comb_2048 0x00102 - 0 - 2048 - - - - 0 - - - -
zio_buf_comb_2560 0x00102 - 2560 - 2560 - - - - 1 - - - -
zio_buf_comb_3072 0x00102 - 0 - 3072 - - - - 0 - - - -
zio_buf_comb_3584 0x00102 - 0 - 3584 - - - - 0 - - - -
zio_buf_comb_4096 0x00102 - 12288 - 4096 - - - - 3 - - - -
zio_buf_comb_5120 0x00102 - 0 - 5120 - - - - 0 - - - -
zio_buf_comb_6144 0x00102 - 0 - 6144 - - - - 0 - - - -
zio_buf_comb_7168 0x00102 - 0 - 7168 - - - - 0 - - - -
zio_buf_comb_8192 0x00102 - 0 - 8192 - - - - 0 - - - -
zio_buf_comb_10240 0x00102 - 0 - 10240 - - - - 0 - - - -
zio_buf_comb_12288 0x00102 - 0 - 12288 - - - - 0 - - - -
zio_buf_comb_14336 0x00102 - 14336 - 14336 - - - - 1 - - - -
zio_buf_comb_16384 0x00102 - 671744 - 16384 - - - - 41 - - - -
zio_buf_comb_20480 0x00082 401408 163840 200704 20480 2 1 1 16 8 8 0 0 0
zio_buf_comb_24576 0x00082 2334720 1769472 233472 24576 10 9 9 80 72 72 0 0 0
zio_buf_comb_28672 0x00082 532480 229376 266240 28672 2 1 1 16 8 8 0 0 0
zio_buf_comb_32768 0x00082 598016 262144 299008 32768 2 1 1 16 8 8 0 0 0
zio_buf_comb_40960 0x00082 4009984 3276800 364544 40960 11 10 10 88 80 80 0 0 0
zio_buf_comb_49152 0x00082 3870720 3145728 430080 49152 9 8 8 72 64 64 0 0 0
zio_buf_comb_57344 0x00082 3964928 3211264 495616 57344 8 7 7 64 56 56 0 0 0
zio_buf_comb_65536 0x00082 1122304 524288 561152 65536 2 1 1 16 8 8 0 0 0
zio_buf_comb_81920 0x00082 8998912 7864320 692224 81920 13 12 12 104 96 96 0 0 0
zio_buf_comb_98304 0x00082 10702848 9437184 823296 98304 13 12 12 104 96 96 0 0 0
zio_buf_comb_114688 0x00082 8589312 7340032 954368 114688 9 8 8 72 64 64 0 0 0
zio_buf_comb_131072 0x00082 106373120 73400320 1085440 131072 98 98 192 784 560 1536 0 0 0
zio_buf_comb_163840 0x00082 0 0 1347584 163840 0 0 0 0 0 0 0 0 0
zio_buf_comb_196608 0x00082 0 0 1609728 196608 0 0 0 0 0 0 0 0 0
zio_buf_comb_229376 0x00082 0 0 1871872 229376 0 0 0 0 0 0 0 0 0
zio_buf_comb_262144 0x00082 0 0 2134016 262144 0 0 0 0 0 0 0 0 0
zio_buf_comb_327680 0x00082 0 0 2658304 327680 0 0 0 0 0 0 0 0 0
zio_buf_comb_393216 0x00082 0 0 3182592 393216 0 0 0 0 0 0 0 0 0
zio_buf_comb_458752 0x00082 0 0 3706880 458752 0 0 0 0 0 0 0 0 0
zio_buf_comb_524288 0x00082 0 0 4231168 524288 0 0 0 0 0 0 0 0 0
zio_buf_comb_655360 0x00082 0 0 5279744 655360 0 0 0 0 0 0 0 0 0
zio_buf_comb_786432 0x00082 0 0 6328320 786432 0 0 0 0 0 0 0 0 0
zio_buf_comb_917504 0x00082 0 0 7376896 917504 0 0 0 0 0 0 0 0 0
zio_buf_comb_1048576 0x00082 8425472 4194304 8425472 1048576 1 1 1 8 4 4 0 0 0
zio_buf_comb_1310720 0x00082 0 0 10522624 1310720 0 0 0 0 0 0 0 0 0
zio_buf_comb_1572864 0x00082 0 0 12619776 1572864 0 0 0 0 0 0 0 0 0
zio_buf_comb_1835008 0x00082 0 0 14716928 1835008 0 0 0 0 0 0 0 0 0
zio_buf_comb_2097152 0x00082 0 0 16814080 2097152 0 0 0 0 0 0 0 0 0
zio_buf_comb_2621440 0x00082 0 0 21008384 2621440 0 0 0 0 0 0 0 0 0
zio_buf_comb_3145728 0x00082 0 0 25202688 3145728 0 0 0 0 0 0 0 0 0
zio_buf_comb_3670016 0x00082 0 0 29396992 3670016 0 0 0 0 0 0 0 0 0
zio_buf_comb_4194304 0x00082 0 0 29392896 4194304 0 0 0 0 0 0 0 0 0
zio_buf_comb_5242880 0x00082 0 0 31485952 5242880 0 0 0 0 0 0 0 0 0
zio_buf_comb_6291456 0x00082 0 0 31481856 6291456 0 0 0 0 0 0 0 0 0
zio_buf_comb_7340032 0x00082 0 0 29380608 7340032 0 0 0 0 0 0 0 0 0
zio_buf_comb_8388608 0x00082 0 0 25182208 8388608 0 0 0 0 0 0 0 0 0
zio_buf_comb_10485760 0x00082 0 0 31473664 10485760 0 0 0 0 0 0 0 0 0
zio_buf_comb_12582912 0x00082 0 0 25178112 12582912 0 0 0 0 0 0 0 0 0
zio_buf_comb_14680064 0x00082 0 0 29372416 14680064 0 0 0 0 0 0 0 0 0
zio_buf_comb_16777216 0x00082 0 0 16785408 16777216 0 0 0 0 0 0 0 0 0
lz4_cache 0x00100 - 0 - 16384 - - - - 0 - - - -
abd_t 0x00100 - 1531192 - 104 - - - - 14723 - - - -
sa_cache 0x00100 - 3472 - 248 - - - - 14 - - - -
dnode_t 0x00100 - 367032 - 984 - - - - 373 - - - -
arc_buf_hdr_t_full 0x00100 - 3536160 - 240 - - - - 14734 - - - -
arc_buf_hdr_t_l2only 0x00100 - 0 - 96 - - - - 0 - - - -
arc_buf_t 0x00100 - 16640 - 32 - - - - 520 - - - -
dmu_buf_impl_t 0x00100 - 299880 - 392 - - - - 765 - - - -
zil_lwb_cache 0x00100 - 0 - 408 - - - - 0 - - - -
zil_zcw_cache 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_0 0x00100 - 0 - 136 - - - - 0 - - - -
sio_cache_1 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_2 0x00100 - 0 - 168 - - - - 0 - - - -
zfs_znode_cache 0x00100 - 18176 - 1136 - - - - 16 - - - -
zfs_znode_hold_cache 0x00100 - 0 - 88 - - - - 0 - - - -
import pool zp-erp1-hdd
root@erpband-pve1:~# zpool import -d /dev/disk/by-partlabel/ zp-erp1-hdd
root@erpband-pve1:~# cat /proc/spl/kmem/slab
--------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency ---
name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max
spl_zlib_workspace_cache 0x00080 0 0 2145216 268104 0 0 0 0 0 0 0 0 0
kcf_context_cache 0x00100 - 0 - 40 - - - - 0 - - - -
zfs_btree_leaf_cache 0x00100 - 33320960 - 4096 - - - - 8135 - - - -
metaslab_alloc_trace_cache 0x00100 - 0 - 72 - - - - 0 - - - -
brt_entry_cache 0x00100 - 0 - 40 - - - - 0 - - - -
brt_pending_entry_cache 0x00100 - 0 - 160 - - - - 0 - - - -
ddt_cache 0x00080 1791936 1589760 199104 24840 9 8 8 72 64 64 0 0 0
ddt_entry_cache 0x00100 - 41959680 - 448 - - - - 93660 - - - -
zio_cache 0x00100 - 1708556800 - 1280 - - - - 1334810 - - - -
zio_link_cache 0x00100 - 64067040 - 48 - - - - 1334730 - - - -
zio_buf_comb_512 0x00102 - 28160 - 512 - - - - 55 - - - -
zio_buf_comb_1024 0x00102 - 0 - 1024 - - - - 0 - - - -
zio_buf_comb_1536 0x00102 - 6144 - 1536 - - - - 4 - - - -
zio_buf_comb_2048 0x00102 - 0 - 2048 - - - - 0 - - - -
zio_buf_comb_2560 0x00102 - 5120 - 2560 - - - - 2 - - - -
zio_buf_comb_3072 0x00102 - 0 - 3072 - - - - 0 - - - -
zio_buf_comb_3584 0x00102 - 0 - 3584 - - - - 0 - - - -
zio_buf_comb_4096 0x00102 - 32768 - 4096 - - - - 8 - - - -
zio_buf_comb_5120 0x00102 - 0 - 5120 - - - - 0 - - - -
zio_buf_comb_6144 0x00102 - 0 - 6144 - - - - 0 - - - -
zio_buf_comb_7168 0x00102 - 0 - 7168 - - - - 0 - - - -
zio_buf_comb_8192 0x00102 - 0 - 8192 - - - - 0 - - - -
zio_buf_comb_10240 0x00102 - 0 - 10240 - - - - 0 - - - -
zio_buf_comb_12288 0x00102 - 0 - 12288 - - - - 0 - - - -
zio_buf_comb_14336 0x00102 - 14336 - 14336 - - - - 1 - - - -
zio_buf_comb_16384 0x00102 - 18563072 - 16384 - - - - 1133 - - - -
zio_buf_comb_20480 0x00082 401408 163840 200704 20480 2 1 1 16 8 8 0 0 0
zio_buf_comb_24576 0x00082 3035136 2359296 233472 24576 13 12 12 104 96 96 0 0 0
zio_buf_comb_28672 0x00082 532480 229376 266240 28672 2 1 1 16 8 8 0 0 0
zio_buf_comb_32768 0x00082 118407168 99614720 299008 32768 396 396 398 3168 3040 3184 0 0 0
zio_buf_comb_40960 0x00082 4009984 3276800 364544 40960 11 10 10 88 80 80 0 0 0
zio_buf_comb_49152 0x00082 5591040 4718592 430080 49152 13 12 12 104 96 96 0 0 0
zio_buf_comb_57344 0x00082 6443008 5505024 495616 57344 13 12 12 104 96 96 0 0 0
zio_buf_comb_65536 0x00082 1122304 524288 561152 65536 2 1 1 16 8 8 0 0 0
zio_buf_comb_81920 0x00082 8998912 7864320 692224 81920 13 12 12 104 96 96 0 0 0
zio_buf_comb_98304 0x00082 10702848 9437184 823296 98304 13 12 12 104 96 96 0 0 0
zio_buf_comb_114688 0x00082 20996096 19267584 954368 114688 22 21 21 176 168 168 0 0 0
zio_buf_comb_131072 0x00082 74895360 16777216 1085440 131072 69 69 636 552 128 5088 0 0 0
zio_buf_comb_163840 0x00082 0 0 1347584 163840 0 0 0 0 0 0 0 0 0
zio_buf_comb_196608 0x00082 0 0 1609728 196608 0 0 0 0 0 0 0 0 0
zio_buf_comb_229376 0x00082 0 0 1871872 229376 0 0 0 0 0 0 0 0 0
zio_buf_comb_262144 0x00082 0 0 2134016 262144 0 0 0 0 0 0 0 0 0
zio_buf_comb_327680 0x00082 0 0 2658304 327680 0 0 0 0 0 0 0 0 0
zio_buf_comb_393216 0x00082 0 0 3182592 393216 0 0 0 0 0 0 0 0 0
zio_buf_comb_458752 0x00082 0 0 3706880 458752 0 0 0 0 0 0 0 0 0
zio_buf_comb_524288 0x00082 0 0 4231168 524288 0 0 0 0 0 0 0 0 0
zio_buf_comb_655360 0x00082 0 0 5279744 655360 0 0 0 0 0 0 0 0 0
zio_buf_comb_786432 0x00082 0 0 6328320 786432 0 0 0 0 0 0 0 0 0
zio_buf_comb_917504 0x00082 0 0 7376896 917504 0 0 0 0 0 0 0 0 0
zio_buf_comb_1048576 0x00082 8425472 4194304 8425472 1048576 1 1 1 8 4 4 0 0 0
zio_buf_comb_1310720 0x00082 0 0 10522624 1310720 0 0 0 0 0 0 0 0 0
zio_buf_comb_1572864 0x00082 0 0 12619776 1572864 0 0 0 0 0 0 0 0 0
zio_buf_comb_1835008 0x00082 0 0 14716928 1835008 0 0 0 0 0 0 0 0 0
zio_buf_comb_2097152 0x00082 0 0 16814080 2097152 0 0 0 0 0 0 0 0 0
zio_buf_comb_2621440 0x00082 0 0 21008384 2621440 0 0 0 0 0 0 0 0 0
zio_buf_comb_3145728 0x00082 0 0 25202688 3145728 0 0 0 0 0 0 0 0 0
zio_buf_comb_3670016 0x00082 0 0 29396992 3670016 0 0 0 0 0 0 0 0 0
zio_buf_comb_4194304 0x00082 0 0 29392896 4194304 0 0 0 0 0 0 0 0 0
zio_buf_comb_5242880 0x00082 0 0 31485952 5242880 0 0 0 0 0 0 0 0 0
zio_buf_comb_6291456 0x00082 0 0 31481856 6291456 0 0 0 0 0 0 0 0 0
zio_buf_comb_7340032 0x00082 0 0 29380608 7340032 0 0 0 0 0 0 0 0 0
zio_buf_comb_8388608 0x00082 0 0 25182208 8388608 0 0 0 0 0 0 0 0 0
zio_buf_comb_10485760 0x00082 0 0 31473664 10485760 0 0 0 0 0 0 0 0 0
zio_buf_comb_12582912 0x00082 0 0 25178112 12582912 0 0 0 0 0 0 0 0 0
zio_buf_comb_14680064 0x00082 0 0 29372416 14680064 0 0 0 0 0 0 0 0 0
zio_buf_comb_16777216 0x00082 0 0 16785408 16777216 0 0 0 0 0 0 0 0 0
lz4_cache 0x00100 - 0 - 16384 - - - - 0 - - - -
abd_t 0x00100 - 5090176 - 104 - - - - 48944 - - - -
sa_cache 0x00100 - 2976 - 248 - - - - 12 - - - -
dnode_t 0x00100 - 3540432 - 984 - - - - 3598 - - - -
arc_buf_hdr_t_full 0x00100 - 20352720 - 240 - - - - 84803 - - - -
arc_buf_hdr_t_l2only 0x00100 - 0 - 96 - - - - 0 - - - -
arc_buf_t 0x00100 - 131168 - 32 - - - - 4099 - - - -
dmu_buf_impl_t 0x00100 - 2934120 - 392 - - - - 7485 - - - -
zil_lwb_cache 0x00100 - 0 - 408 - - - - 0 - - - -
zil_zcw_cache 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_0 0x00100 - 0 - 136 - - - - 0 - - - -
sio_cache_1 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_2 0x00100 - 0 - 168 - - - - 0 - - - -
zfs_znode_cache 0x00100 - 20448 - 1136 - - - - 18 - - - -
zfs_znode_hold_cache 0x00100 - 0 - 88 - - - - 0 - - - -
3.3 GiB - ARC size (current) 3.0 GiB - Anonymous metadata size
root@erpband-pve1:~# 3gb anon
-bash: 3gb: command not found
root@erpband-pve1:~#
root@erpband-pve1:~# cat /proc/spl/kmem/slab
--------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency ---
name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max
spl_zlib_workspace_cache 0x00080 0 0 2145216 268104 0 0 0 0 0 0 0 0 0
kcf_context_cache 0x00100 - 0 - 40 - - - - 0 - - - -
zfs_btree_leaf_cache 0x00100 - 34217984 - 4096 - - - - 8354 - - - -
metaslab_alloc_trace_cache 0x00100 - 0 - 72 - - - - 0 - - - -
brt_entry_cache 0x00100 - 0 - 40 - - - - 0 - - - -
brt_pending_entry_cache 0x00100 - 0 - 160 - - - - 0 - - - -
ddt_cache 0x00080 1791936 1589760 199104 24840 9 8 8 72 64 64 0 0 0
ddt_entry_cache 0x00100 - 2141819904 - 448 - - - - 4780848 - - - -
zio_cache 0x00100 - 98560 - 1280 - - - - 77 - - - -
zio_link_cache 0x00100 - 144 - 48 - - - - 3 - - - -
zio_buf_comb_512 0x00102 - 28160 - 512 - - - - 55 - - - -
zio_buf_comb_1024 0x00102 - 0 - 1024 - - - - 0 - - - -
zio_buf_comb_1536 0x00102 - 6144 - 1536 - - - - 4 - - - -
zio_buf_comb_2048 0x00102 - 0 - 2048 - - - - 0 - - - -
zio_buf_comb_2560 0x00102 - 5120 - 2560 - - - - 2 - - - -
zio_buf_comb_3072 0x00102 - 0 - 3072 - - - - 0 - - - -
zio_buf_comb_3584 0x00102 - 0 - 3584 - - - - 0 - - - -
zio_buf_comb_4096 0x00102 - 32768 - 4096 - - - - 8 - - - -
zio_buf_comb_5120 0x00102 - 0 - 5120 - - - - 0 - - - -
zio_buf_comb_6144 0x00102 - 0 - 6144 - - - - 0 - - - -
zio_buf_comb_7168 0x00102 - 0 - 7168 - - - - 0 - - - -
zio_buf_comb_8192 0x00102 - 0 - 8192 - - - - 0 - - - -
zio_buf_comb_10240 0x00102 - 0 - 10240 - - - - 0 - - - -
zio_buf_comb_12288 0x00102 - 0 - 12288 - - - - 0 - - - -
zio_buf_comb_14336 0x00102 - 14336 - 14336 - - - - 1 - - - -
zio_buf_comb_16384 0x00102 - 18563072 - 16384 - - - - 1133 - - - -
zio_buf_comb_20480 0x00082 401408 163840 200704 20480 2 1 1 16 8 8 0 0 0
zio_buf_comb_24576 0x00082 3035136 2359296 233472 24576 13 12 12 104 96 96 0 0 0
zio_buf_comb_28672 0x00082 532480 229376 266240 28672 2 1 1 16 8 8 0 0 0
zio_buf_comb_32768 0x00082 3045695488 2669936640 299008 32768 10186 10185 10185 81488 81480 81480 0 0 0
zio_buf_comb_40960 0x00082 4009984 3276800 364544 40960 11 10 10 88 80 80 0 0 0
zio_buf_comb_49152 0x00082 5591040 4718592 430080 49152 13 12 12 104 96 96 0 0 0
zio_buf_comb_57344 0x00082 6443008 5505024 495616 57344 13 12 12 104 96 96 0 0 0
zio_buf_comb_65536 0x00082 1122304 524288 561152 65536 2 1 1 16 8 8 0 0 0
zio_buf_comb_81920 0x00082 8998912 7864320 692224 81920 13 12 12 104 96 96 0 0 0
zio_buf_comb_98304 0x00082 10702848 9437184 823296 98304 13 12 12 104 96 96 0 0 0
zio_buf_comb_114688 0x00082 20996096 19267584 954368 114688 22 21 21 176 168 168 0 0 0
zio_buf_comb_131072 0x00082 74895360 16777216 1085440 131072 69 69 636 552 128 5088 0 0 0
zio_buf_comb_163840 0x00082 0 0 1347584 163840 0 0 0 0 0 0 0 0 0
zio_buf_comb_196608 0x00082 0 0 1609728 196608 0 0 0 0 0 0 0 0 0
zio_buf_comb_229376 0x00082 0 0 1871872 229376 0 0 0 0 0 0 0 0 0
zio_buf_comb_262144 0x00082 0 0 2134016 262144 0 0 0 0 0 0 0 0 0
zio_buf_comb_327680 0x00082 0 0 2658304 327680 0 0 0 0 0 0 0 0 0
zio_buf_comb_393216 0x00082 0 0 3182592 393216 0 0 0 0 0 0 0 0 0
zio_buf_comb_458752 0x00082 0 0 3706880 458752 0 0 0 0 0 0 0 0 0
zio_buf_comb_524288 0x00082 0 0 4231168 524288 0 0 0 0 0 0 0 0 0
zio_buf_comb_655360 0x00082 0 0 5279744 655360 0 0 0 0 0 0 0 0 0
zio_buf_comb_786432 0x00082 0 0 6328320 786432 0 0 0 0 0 0 0 0 0
zio_buf_comb_917504 0x00082 0 0 7376896 917504 0 0 0 0 0 0 0 0 0
zio_buf_comb_1048576 0x00082 8425472 4194304 8425472 1048576 1 1 1 8 4 4 0 0 0
zio_buf_comb_1310720 0x00082 0 0 10522624 1310720 0 0 0 0 0 0 0 0 0
zio_buf_comb_1572864 0x00082 0 0 12619776 1572864 0 0 0 0 0 0 0 0 0
zio_buf_comb_1835008 0x00082 0 0 14716928 1835008 0 0 0 0 0 0 0 0 0
zio_buf_comb_2097152 0x00082 0 0 16814080 2097152 0 0 0 0 0 0 0 0 0
zio_buf_comb_2621440 0x00082 0 0 21008384 2621440 0 0 0 0 0 0 0 0 0
zio_buf_comb_3145728 0x00082 0 0 25202688 3145728 0 0 0 0 0 0 0 0 0
zio_buf_comb_3670016 0x00082 0 0 29396992 3670016 0 0 0 0 0 0 0 0 0
zio_buf_comb_4194304 0x00082 0 0 29392896 4194304 0 0 0 0 0 0 0 0 0
zio_buf_comb_5242880 0x00082 0 0 31485952 5242880 0 0 0 0 0 0 0 0 0
zio_buf_comb_6291456 0x00082 0 0 31481856 6291456 0 0 0 0 0 0 0 0 0
zio_buf_comb_7340032 0x00082 0 0 29380608 7340032 0 0 0 0 0 0 0 0 0
zio_buf_comb_8388608 0x00082 0 0 25182208 8388608 0 0 0 0 0 0 0 0 0
zio_buf_comb_10485760 0x00082 0 0 31473664 10485760 0 0 0 0 0 0 0 0 0
zio_buf_comb_12582912 0x00082 0 0 25178112 12582912 0 0 0 0 0 0 0 0 0
zio_buf_comb_14680064 0x00082 0 0 29372416 14680064 0 0 0 0 0 0 0 0 0
zio_buf_comb_16777216 0x00082 0 0 16785408 16777216 0 0 0 0 0 0 0 0 0
lz4_cache 0x00100 - 0 - 16384 - - - - 0 - - - -
abd_t 0x00100 - 8599240 - 104 - - - - 82685 - - - -
sa_cache 0x00100 - 2976 - 248 - - - - 12 - - - -
dnode_t 0x00100 - 3540432 - 984 - - - - 3598 - - - -
arc_buf_hdr_t_full 0x00100 - 29341680 - 240 - - - - 122257 - - - -
arc_buf_hdr_t_l2only 0x00100 - 0 - 96 - - - - 0 - - - -
arc_buf_t 0x00100 - 2645856 - 32 - - - - 82683 - - - -
dmu_buf_impl_t 0x00100 - 33736696 - 392 - - - - 86063 - - - -
zil_lwb_cache 0x00100 - 0 - 408 - - - - 0 - - - -
zil_zcw_cache 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_0 0x00100 - 0 - 136 - - - - 0 - - - -
sio_cache_1 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_2 0x00100 - 0 - 168 - - - - 0 - - - -
zfs_znode_cache 0x00100 - 20448 - 1136 - - - - 18 - - - -
zfs_znode_hold_cache 0x00100 - 0 - 88 - - - - 0 - - - -
20.6 GiB - ARC size (current) 20.1 GiB - Anonymous metadata size
root@erpband-pve1:~# cat /proc/spl/kmem/slab
--------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency ---
name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max
spl_zlib_workspace_cache 0x00080 0 0 2145216 268104 0 0 0 0 0 0 0 0 0
kcf_context_cache 0x00100 - 0 - 40 - - - - 0 - - - -
zfs_btree_leaf_cache 0x00100 - 40005632 - 4096 - - - - 9767 - - - -
metaslab_alloc_trace_cache 0x00100 - 0 - 72 - - - - 0 - - - -
brt_entry_cache 0x00100 - 0 - 40 - - - - 0 - - - -
brt_pending_entry_cache 0x00100 - 0 - 160 - - - - 0 - - - -
ddt_cache 0x00080 1791936 1589760 199104 24840 9 8 8 72 64 64 0 0 0
ddt_entry_cache 0x00100 - 1925172480 - 448 - - - - 4297260 - - - -
zio_cache 0x00100 - 98560 - 1280 - - - - 77 - - - -
zio_link_cache 0x00100 - 144 - 48 - - - - 3 - - - -
zio_buf_comb_512 0x00102 - 28160 - 512 - - - - 55 - - - -
zio_buf_comb_1024 0x00102 - 0 - 1024 - - - - 0 - - - -
zio_buf_comb_1536 0x00102 - 6144 - 1536 - - - - 4 - - - -
zio_buf_comb_2048 0x00102 - 0 - 2048 - - - - 0 - - - -
zio_buf_comb_2560 0x00102 - 5120 - 2560 - - - - 2 - - - -
zio_buf_comb_3072 0x00102 - 0 - 3072 - - - - 0 - - - -
zio_buf_comb_3584 0x00102 - 0 - 3584 - - - - 0 - - - -
zio_buf_comb_4096 0x00102 - 32768 - 4096 - - - - 8 - - - -
zio_buf_comb_5120 0x00102 - 0 - 5120 - - - - 0 - - - -
zio_buf_comb_6144 0x00102 - 0 - 6144 - - - - 0 - - - -
zio_buf_comb_7168 0x00102 - 0 - 7168 - - - - 0 - - - -
zio_buf_comb_8192 0x00102 - 0 - 8192 - - - - 0 - - - -
zio_buf_comb_10240 0x00102 - 0 - 10240 - - - - 0 - - - -
zio_buf_comb_12288 0x00102 - 0 - 12288 - - - - 0 - - - -
zio_buf_comb_14336 0x00102 - 14336 - 14336 - - - - 1 - - - -
zio_buf_comb_16384 0x00102 - 18563072 - 16384 - - - - 1133 - - - -
zio_buf_comb_20480 0x00082 401408 163840 200704 20480 2 1 1 16 8 8 0 0 0
zio_buf_comb_24576 0x00082 3035136 2359296 233472 24576 13 12 12 104 96 96 0 0 0
zio_buf_comb_28672 0x00082 532480 229376 266240 28672 2 1 1 16 8 8 0 0 0
zio_buf_comb_32768 0x00082 16854183936 14776008704 299008 32768 56367 56366 56366 450936 450928 450928 0 0 0
zio_buf_comb_40960 0x00082 4009984 3276800 364544 40960 11 10 10 88 80 80 0 0 0
zio_buf_comb_49152 0x00082 5591040 4718592 430080 49152 13 12 12 104 96 96 0 0 0
zio_buf_comb_57344 0x00082 6443008 5505024 495616 57344 13 12 12 104 96 96 0 0 0
zio_buf_comb_65536 0x00082 1122304 524288 561152 65536 2 1 1 16 8 8 0 0 0
zio_buf_comb_81920 0x00082 8998912 7864320 692224 81920 13 12 12 104 96 96 0 0 0
zio_buf_comb_98304 0x00082 10702848 9437184 823296 98304 13 12 12 104 96 96 0 0 0
zio_buf_comb_114688 0x00082 20996096 19267584 954368 114688 22 21 21 176 168 168 0 0 0
zio_buf_comb_131072 0x00082 74895360 16777216 1085440 131072 69 69 636 552 128 5088 0 0 0
zio_buf_comb_163840 0x00082 0 0 1347584 163840 0 0 0 0 0 0 0 0 0
zio_buf_comb_196608 0x00082 0 0 1609728 196608 0 0 0 0 0 0 0 0 0
zio_buf_comb_229376 0x00082 0 0 1871872 229376 0 0 0 0 0 0 0 0 0
zio_buf_comb_262144 0x00082 0 0 2134016 262144 0 0 0 0 0 0 0 0 0
zio_buf_comb_327680 0x00082 0 0 2658304 327680 0 0 0 0 0 0 0 0 0
zio_buf_comb_393216 0x00082 0 0 3182592 393216 0 0 0 0 0 0 0 0 0
zio_buf_comb_458752 0x00082 0 0 3706880 458752 0 0 0 0 0 0 0 0 0
zio_buf_comb_524288 0x00082 0 0 4231168 524288 0 0 0 0 0 0 0 0 0
zio_buf_comb_655360 0x00082 0 0 5279744 655360 0 0 0 0 0 0 0 0 0
zio_buf_comb_786432 0x00082 0 0 6328320 786432 0 0 0 0 0 0 0 0 0
zio_buf_comb_917504 0x00082 0 0 7376896 917504 0 0 0 0 0 0 0 0 0
zio_buf_comb_1048576 0x00082 8425472 4194304 8425472 1048576 1 1 1 8 4 4 0 0 0
zio_buf_comb_1310720 0x00082 0 0 10522624 1310720 0 0 0 0 0 0 0 0 0
zio_buf_comb_1572864 0x00082 0 0 12619776 1572864 0 0 0 0 0 0 0 0 0
zio_buf_comb_1835008 0x00082 0 0 14716928 1835008 0 0 0 0 0 0 0 0 0
zio_buf_comb_2097152 0x00082 0 0 16814080 2097152 0 0 0 0 0 0 0 0 0
zio_buf_comb_2621440 0x00082 0 0 21008384 2621440 0 0 0 0 0 0 0 0 0
zio_buf_comb_3145728 0x00082 0 0 25202688 3145728 0 0 0 0 0 0 0 0 0
zio_buf_comb_3670016 0x00082 0 0 29396992 3670016 0 0 0 0 0 0 0 0 0
zio_buf_comb_4194304 0x00082 0 0 29392896 4194304 0 0 0 0 0 0 0 0 0
zio_buf_comb_5242880 0x00082 0 0 31485952 5242880 0 0 0 0 0 0 0 0 0
zio_buf_comb_6291456 0x00082 0 0 31481856 6291456 0 0 0 0 0 0 0 0 0
zio_buf_comb_7340032 0x00082 0 0 29380608 7340032 0 0 0 0 0 0 0 0 0
zio_buf_comb_8388608 0x00082 0 0 25182208 8388608 0 0 0 0 0 0 0 0 0
zio_buf_comb_10485760 0x00082 0 0 31473664 10485760 0 0 0 0 0 0 0 0 0
zio_buf_comb_12582912 0x00082 0 0 25178112 12582912 0 0 0 0 0 0 0 0 0
zio_buf_comb_14680064 0x00082 0 0 29372416 14680064 0 0 0 0 0 0 0 0 0
zio_buf_comb_16777216 0x00082 0 0 16785408 16777216 0 0 0 0 0 0 0 0 0
lz4_cache 0x00100 - 0 - 16384 - - - - 0 - - - -
abd_t 0x00100 - 47021936 - 104 - - - - 452134 - - - -
sa_cache 0x00100 - 2976 - 248 - - - - 12 - - - -
dnode_t 0x00100 - 3540432 - 984 - - - - 3598 - - - -
arc_buf_hdr_t_full 0x00100 - 116441760 - 240 - - - - 485174 - - - -
arc_buf_hdr_t_l2only 0x00100 - 0 - 96 - - - - 0 - - - -
arc_buf_t 0x00100 - 14468224 - 32 - - - - 452132 - - - -
dmu_buf_impl_t 0x00100 - 178560704 - 392 - - - - 455512 - - - -
zil_lwb_cache 0x00100 - 0 - 408 - - - - 0 - - - -
zil_zcw_cache 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_0 0x00100 - 0 - 136 - - - - 0 - - - -
sio_cache_1 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_2 0x00100 - 0 - 168 - - - - 0 - - - -
zfs_znode_cache 0x00100 - 20448 - 1136 - - - - 18 - - - -
zfs_znode_hold_cache 0x00100 - 0 - 88 - - - - 0 - - - -
51.1 GiB - ARC size (current) 50.2 GiB - Anonymous metadata size
root@erpband-pve1:~# cat /proc/spl/kmem/slab
--------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency ---
name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max
spl_zlib_workspace_cache 0x00080 0 0 2145216 268104 0 0 0 0 0 0 0 0 0
kcf_context_cache 0x00100 - 0 - 40 - - - - 0 - - - -
zfs_btree_leaf_cache 0x00100 - 47239168 - 4096 - - - - 11533 - - - -
metaslab_alloc_trace_cache 0x00100 - 0 - 72 - - - - 0 - - - -
brt_entry_cache 0x00100 - 0 - 40 - - - - 0 - - - -
brt_pending_entry_cache 0x00100 - 0 - 160 - - - - 0 - - - -
ddt_cache 0x00080 1791936 1589760 199104 24840 9 8 8 72 64 64 0 0 0
ddt_entry_cache 0x00100 - 857299072 - 448 - - - - 1913614 - - - -
zio_cache 0x00100 - 93440 - 1280 - - - - 73 - - - -
zio_link_cache 0x00100 - 0 - 48 - - - - 0 - - - -
zio_buf_comb_512 0x00102 - 28160 - 512 - - - - 55 - - - -
zio_buf_comb_1024 0x00102 - 0 - 1024 - - - - 0 - - - -
zio_buf_comb_1536 0x00102 - 6144 - 1536 - - - - 4 - - - -
zio_buf_comb_2048 0x00102 - 0 - 2048 - - - - 0 - - - -
zio_buf_comb_2560 0x00102 - 5120 - 2560 - - - - 2 - - - -
zio_buf_comb_3072 0x00102 - 0 - 3072 - - - - 0 - - - -
zio_buf_comb_3584 0x00102 - 0 - 3584 - - - - 0 - - - -
zio_buf_comb_4096 0x00102 - 32768 - 4096 - - - - 8 - - - -
zio_buf_comb_5120 0x00102 - 0 - 5120 - - - - 0 - - - -
zio_buf_comb_6144 0x00102 - 0 - 6144 - - - - 0 - - - -
zio_buf_comb_7168 0x00102 - 0 - 7168 - - - - 0 - - - -
zio_buf_comb_8192 0x00102 - 0 - 8192 - - - - 0 - - - -
zio_buf_comb_10240 0x00102 - 0 - 10240 - - - - 0 - - - -
zio_buf_comb_12288 0x00102 - 0 - 12288 - - - - 0 - - - -
zio_buf_comb_14336 0x00102 - 14336 - 14336 - - - - 1 - - - -
zio_buf_comb_16384 0x00102 - 18563072 - 16384 - - - - 1133 - - - -
zio_buf_comb_20480 0x00082 200704 163840 200704 20480 1 1 1 8 8 8 0 0 0
zio_buf_comb_24576 0x00082 2568192 2162688 233472 24576 11 11 12 88 88 96 0 0 0
zio_buf_comb_28672 0x00082 266240 229376 266240 28672 1 1 1 8 8 8 0 0 0
zio_buf_comb_32768 0x00082 38581002240 33824178176 299008 32768 129030 129029 129029 1032240 1032232 1032232 0 0 0
zio_buf_comb_40960 0x00082 3280896 2908160 364544 40960 9 9 10 72 71 80 0 0 0
zio_buf_comb_49152 0x00082 4730880 4325376 430080 49152 11 11 12 88 88 96 0 0 0
zio_buf_comb_57344 0x00082 5451776 5046272 495616 57344 11 11 12 88 88 96 0 0 0
zio_buf_comb_65536 0x00082 561152 524288 561152 65536 1 1 1 8 8 8 0 0 0
zio_buf_comb_81920 0x00082 7614464 7208960 692224 81920 11 11 12 88 88 96 0 0 0
zio_buf_comb_98304 0x00082 9056256 8650752 823296 98304 11 11 12 88 88 96 0 0 0
zio_buf_comb_114688 0x00082 17178624 14106624 954368 114688 18 18 21 144 123 168 0 0 0
zio_buf_comb_131072 0x00082 66211840 15073280 1085440 131072 61 61 636 488 115 5088 0 0 0
zio_buf_comb_163840 0x00082 0 0 1347584 163840 0 0 0 0 0 0 0 0 0
zio_buf_comb_196608 0x00082 0 0 1609728 196608 0 0 0 0 0 0 0 0 0
zio_buf_comb_229376 0x00082 0 0 1871872 229376 0 0 0 0 0 0 0 0 0
zio_buf_comb_262144 0x00082 0 0 2134016 262144 0 0 0 0 0 0 0 0 0
zio_buf_comb_327680 0x00082 0 0 2658304 327680 0 0 0 0 0 0 0 0 0
zio_buf_comb_393216 0x00082 0 0 3182592 393216 0 0 0 0 0 0 0 0 0
zio_buf_comb_458752 0x00082 0 0 3706880 458752 0 0 0 0 0 0 0 0 0
zio_buf_comb_524288 0x00082 0 0 4231168 524288 0 0 0 0 0 0 0 0 0
zio_buf_comb_655360 0x00082 0 0 5279744 655360 0 0 0 0 0 0 0 0 0
zio_buf_comb_786432 0x00082 0 0 6328320 786432 0 0 0 0 0 0 0 0 0
zio_buf_comb_917504 0x00082 0 0 7376896 917504 0 0 0 0 0 0 0 0 0
zio_buf_comb_1048576 0x00082 8425472 4194304 8425472 1048576 1 1 1 8 4 4 0 0 0
zio_buf_comb_1310720 0x00082 0 0 10522624 1310720 0 0 0 0 0 0 0 0 0
zio_buf_comb_1572864 0x00082 0 0 12619776 1572864 0 0 0 0 0 0 0 0 0
zio_buf_comb_1835008 0x00082 0 0 14716928 1835008 0 0 0 0 0 0 0 0 0
zio_buf_comb_2097152 0x00082 0 0 16814080 2097152 0 0 0 0 0 0 0 0 0
zio_buf_comb_2621440 0x00082 0 0 21008384 2621440 0 0 0 0 0 0 0 0 0
zio_buf_comb_3145728 0x00082 0 0 25202688 3145728 0 0 0 0 0 0 0 0 0
zio_buf_comb_3670016 0x00082 0 0 29396992 3670016 0 0 0 0 0 0 0 0 0
zio_buf_comb_4194304 0x00082 0 0 29392896 4194304 0 0 0 0 0 0 0 0 0
zio_buf_comb_5242880 0x00082 0 0 31485952 5242880 0 0 0 0 0 0 0 0 0
zio_buf_comb_6291456 0x00082 0 0 31481856 6291456 0 0 0 0 0 0 0 0 0
zio_buf_comb_7340032 0x00082 0 0 29380608 7340032 0 0 0 0 0 0 0 0 0
zio_buf_comb_8388608 0x00082 0 0 25182208 8388608 0 0 0 0 0 0 0 0 0
zio_buf_comb_10485760 0x00082 0 0 31473664 10485760 0 0 0 0 0 0 0 0 0
zio_buf_comb_12582912 0x00082 0 0 25178112 12582912 0 0 0 0 0 0 0 0 0
zio_buf_comb_14680064 0x00082 0 0 29372416 14680064 0 0 0 0 0 0 0 0 0
zio_buf_comb_16777216 0x00082 0 0 16785408 16777216 0 0 0 0 0 0 0 0 0
lz4_cache 0x00100 - 0 - 16384 - - - - 0 - - - -
abd_t 0x00100 - 107477760 - 104 - - - - 1033440 - - - -
sa_cache 0x00100 - 2976 - 248 - - - - 12 - - - -
dnode_t 0x00100 - 3540432 - 984 - - - - 3598 - - - -
arc_buf_hdr_t_full 0x00100 - 252282720 - 240 - - - - 1051178 - - - -
arc_buf_hdr_t_l2only 0x00100 - 0 - 96 - - - - 0 - - - -
arc_buf_t 0x00100 - 33070016 - 32 - - - - 1033438 - - - -
dmu_buf_impl_t 0x00100 - 406432656 - 392 - - - - 1036818 - - - -
zil_lwb_cache 0x00100 - 0 - 408 - - - - 0 - - - -
zil_zcw_cache 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_0 0x00100 - 0 - 136 - - - - 0 - - - -
sio_cache_1 0x00100 - 0 - 152 - - - - 0 - - - -
sio_cache_2 0x00100 - 0 - 168 - - - - 0 - - - -
zfs_znode_cache 0x00100 - 20448 - 1136 - - - - 18 - - - -
zfs_znode_hold_cache 0x00100 - 0 - 88 - - - - 0 - - - -
Today I reinstalled proxmox with previous version. It didn' help. The problem is still exist.
Now:
# zfs version
zfs-2.2.3-pve1
zfs-kmod-2.2.3-pve1
was before reinstall
# zfs version
zfs-2.2.4-pve1
zfs-kmod-2.2.4-pve1