zfs
zfs copied to clipboard
openzfs
PANIC: adding existent segment to range tree
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Arch Linux |
| Kernel Version | 6.12.23-1-lts |
| Architecture | x86_64 |
| OpenZFS Version | zfs-2.3.1-1 zfs-kmod-2.3.1-1 (ArchZFS zfs-dkms) |
Describe the problem you're observing
A few weeks ago while building AOSP, I encountered a zfs dataset hang accompanied by a kernel panic and metadata corruption. After the panic, the pool was inaccessible, so I unmounted the datasets with umount -l, rebooted, ran scrub and found that metadata : 0x0 was corrupted. However, the datasets in the pool mounted successfully, and at least the one year old data is not corrupted (I verified this with a backup), and a shallow look at the newer data of an important dataset. The dataset that contains the LOS sources is also seems fine, although it's not as important. What exactly is metadata : 0x0 ? Is only the pool metadata corrupted and the datasets are unaffected? I unfortunately don't have a possibility to fully check this as the last backup was done a year ago.
Describe how to reproduce the problem
At the time of the panic, I was building AOSP, a dataset that contained LOS sources was using zstd compression, dedup, encryption, and running low free space. (Less than 20GB)
Include any warning/errors/backtraces from the system logs
Log of panic
Apr 22 19:37:02 kernel: PANIC: zfs: adding existent segment to range tree (offset=3b5f03e400 size=200)
Apr 22 19:37:02 kernel: Showing stack for process 2062
Apr 22 19:37:02 kernel: CPU: 9 UID: 0 PID: 2062 Comm: txg_sync Tainted: P OE 6.12.23-1-lts #1 5630ad33b0f77f901c9ed98e9ebd46faafd2d56e
Apr 22 19:37:02 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Apr 22 19:37:02 kernel: Call Trace:
Apr 22 19:37:02 kernel: <TASK>
Apr 22 19:37:02 kernel: dump_stack_lvl+0x5d/0x80
Apr 22 19:37:02 kernel: vcmn_err.cold+0x54/0x7f [spl b1a482a702b877ea2b4b0a32d135a7556de94817]
Apr 22 19:37:02 kernel: zfs_panic_recover+0x79/0xa0 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: zfs_range_tree_add_impl+0x2a5/0xfa0 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 22 19:37:02 kernel: ? dmu_zfetch+0x1e/0x60 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: zfs_range_tree_remove_xor_add_segment+0x3a4/0x670 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: zfs_range_tree_remove_xor_add+0x86/0x180 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: metaslab_sync+0x279/0x960 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 22 19:37:02 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 22 19:37:02 kernel: ? dmu_tx_create_dd+0x8f/0xd0 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 22 19:37:02 kernel: vdev_sync+0x72/0x4c0 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 22 19:37:02 kernel: ? spa_flush_metaslabs+0xf1/0x430 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 22 19:37:02 kernel: spa_sync+0x64e/0x1090 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 22 19:37:02 kernel: ? spa_txg_history_init_io+0x115/0x120 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: txg_sync_thread+0x20a/0x3b0 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs e41968296821dc99f1cd0b0142a7e63525970461]
Apr 22 19:37:02 kernel: ? __pfx_thread_generic_wrapper+0x10/0x10 [spl b1a482a702b877ea2b4b0a32d135a7556de94817]
Apr 22 19:37:02 kernel: thread_generic_wrapper+0x5d/0x70 [spl b1a482a702b877ea2b4b0a32d135a7556de94817]
Apr 22 19:37:02 kernel: kthread+0xd2/0x100
Apr 22 19:37:02 kernel: ? __pfx_kthread+0x10/0x10
Apr 22 19:37:02 kernel: ret_from_fork+0x34/0x50
Apr 22 19:37:02 kernel: ? __pfx_kthread+0x10/0x10
Apr 22 19:37:02 kernel: ret_from_fork_asm+0x1a/0x30
Apr 22 19:37:02 kernel: </TASK>
zpool status -v after scrub
pool: sz
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: scrub repaired 0B in 00:01:47 with 7 errors on Tue Apr 22 22:05:28 2025
config:
NAME STATE READ WRITE CKSUM
sz ONLINE 0 0 0
nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6 ONLINE 0 0 0
errors: Permanent errors have been detected in the following files:
<metadata>:<0x0>
zpool events -v
TIME CLASS
Apr 22 2025 21:36:03.751748759 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
history_hostname = "alpha"
history_internal_str = "pool version 5000; software version zfs-2.3.1-0-gf3e4043a3-dist; uts alpha 6.12.23-1-lts #1 SMP PREEMPT_DYNAMIC Thu, 10 Apr 2025 13:28:36 +0000 x86_64"
history_internal_name = "open"
history_txg = 0x1db6aa
history_time = 0x6807c573
time = 0x6807c573 0x2ccec697
eid = 0x1
Apr 22 2025 21:36:03.778412678 sysevent.fs.zfs.config_sync
version = 0x0
class = "sysevent.fs.zfs.config_sync"
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
time = 0x6807c573 0x2e65a286
eid = 0x2
Apr 22 2025 21:36:03.778412678 sysevent.fs.zfs.pool_import
version = 0x0
class = "sysevent.fs.zfs.pool_import"
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
time = 0x6807c573 0x2e65a286
eid = 0x3
Apr 22 2025 21:36:03.781745669 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
history_hostname = "alpha"
history_internal_str = "pool version 5000; software version zfs-2.3.1-0-gf3e4043a3-dist; uts alpha 6.12.23-1-lts #1 SMP PREEMPT_DYNAMIC Thu, 10 Apr 2025 13:28:36 +0000 x86_64"
history_internal_name = "import"
history_txg = 0x1db6ac
history_time = 0x6807c573
time = 0x6807c573 0x2e987e05
eid = 0x4
Apr 22 2025 21:36:03.801743607 sysevent.fs.zfs.config_sync
version = 0x0
class = "sysevent.fs.zfs.config_sync"
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
time = 0x6807c573 0x2fc9a2f7
eid = 0x5
Apr 22 2025 21:52:30.335147620 sysevent.fs.zfs.scrub_start
version = 0x0
class = "sysevent.fs.zfs.scrub_start"
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
time = 0x6807c94e 0x13f9f264
eid = 0x6
Apr 22 2025 21:52:30.335147620 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
history_hostname = "alpha"
history_internal_str = "func=1 mintxg=0 maxtxg=1947505"
history_internal_name = "scan setup"
history_txg = 0x1db771
history_time = 0x6807c94e
time = 0x6807c94e 0x13f9f264
eid = 0x7
Apr 22 2025 21:52:30.465148965 ereport.fs.zfs.data
class = "ereport.fs.zfs.data"
ena = 0x1965edafb9f02c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_err = 0x34
zio_flags = 0x30000b0 [SCRUB SCAN_THREAD CANFAIL RAW_COMPRESS RAW_ENCRYPT]
zio_stage = 0x4000000 [DONE]
zio_pipeline = 0x4100c02 [READ_BP_INIT DDT_READ_START DDT_READ_DONE READY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
time = 0x6807c94e 0x1bb99c25
eid = 0x8
Apr 22 2025 21:52:30.465148965 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1965edafb9f02c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1965ede1c53
vdev_delta_ts = 0x25f1c
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x3
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3080b1 [DONT_AGGREGATE SCRUB SCAN_THREAD CANFAIL IO_RETRY DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x12700
zio_timestamp = 0x1965edbbd37
zio_delta = 0x25f1c
zio_priority = 0x4 [SCRUB]
zio_offset = 0x14d08a5c00
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
time = 0x6807c94e 0x1bb99c25
eid = 0x9
Apr 22 2025 21:52:30.465148965 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1965edafb9f02c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1965edadc87
vdev_delta_ts = 0x11c39
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x1
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x11a9f
zio_timestamp = 0x1965ed9c04e
zio_delta = 0x11c39
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43e2b0e600
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
time = 0x6807c94e 0x1bb99c25
eid = 0xa
Apr 22 2025 21:52:30.608483783 ereport.fs.zfs.data
class = "ereport.fs.zfs.data"
ena = 0x1966769b01902c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_err = 0x34
zio_flags = 0x30000b0 [SCRUB SCAN_THREAD CANFAIL RAW_COMPRESS RAW_ENCRYPT]
zio_stage = 0x4000000 [DONE]
zio_pipeline = 0x4100c02 [READ_BP_INIT DDT_READ_START DDT_READ_DONE READY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
time = 0x6807c94e 0x2444b9c7
eid = 0xb
Apr 22 2025 21:52:30.608483783 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1966769b01902c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1966770727e
vdev_delta_ts = 0x24388
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x6
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3080b1 [DONT_AGGREGATE SCRUB SCAN_THREAD CANFAIL IO_RETRY DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x130e2
zio_timestamp = 0x196676e2ef6
zio_delta = 0x24388
zio_priority = 0x4 [SCRUB]
zio_offset = 0x14d08a5800
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
time = 0x6807c94e 0x2444b9c7
eid = 0xc
Apr 22 2025 21:52:30.608483783 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1966769b01902c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x196676995c5
vdev_delta_ts = 0x1188c
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x4
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x116de
zio_timestamp = 0x19667687d39
zio_delta = 0x1188c
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43e2b0e200
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
time = 0x6807c94e 0x2444b9c7
eid = 0xd
Apr 22 2025 21:52:30.721818290 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1966e1f421e00001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1966e1f26e4
vdev_delta_ts = 0x11bf2
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x7
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x11a44
zio_timestamp = 0x1966e1e0af2
zio_delta = 0x11bf2
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5fa00
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x416
bad_range_clears = 0x3f4
time = 0x6807c94e 0x2b0612b2
eid = 0xe
Apr 22 2025 21:52:30.735151761 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1966edb851402c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1966edb6eee
vdev_delta_ts = 0x11d46
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x9
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x11b8e
zio_timestamp = 0x1966eda51a8
zio_delta = 0x11d46
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5f800
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x3f9
bad_range_clears = 0x424
time = 0x6807c94e 0x2bd18691
eid = 0xf
Apr 22 2025 21:52:30.778485544 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x196717cc4af00c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x196717ca9c5
vdev_delta_ts = 0x12278
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0xb
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x1207a
zio_timestamp = 0x196717b874d
zio_delta = 0x12278
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5f400
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x3fe
bad_range_clears = 0x3e1
time = 0x6807c94e 0x2e66bf28
eid = 0x10
Apr 22 2025 21:52:30.841819533 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x196754093bc00401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x19675407db4
vdev_delta_ts = 0xf3e8
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0xd
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0xf24e
zio_timestamp = 0x196753f89cc
zio_delta = 0xf3e8
zio_priority = 0x4 [SCRUB]
zio_offset = 0x39703f1800
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x410
bad_range_clears = 0x3d3
time = 0x6807c94e 0x322d258d
eid = 0x11
Apr 22 2025 21:52:30.851819638 ereport.fs.zfs.data
class = "ereport.fs.zfs.data"
ena = 0x19675c744f600401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_err = 0x34
zio_flags = 0x30000b0 [SCRUB SCAN_THREAD CANFAIL RAW_COMPRESS RAW_ENCRYPT]
zio_stage = 0x4000000 [DONE]
zio_pipeline = 0x4100c02 [READ_BP_INIT DDT_READ_START DDT_READ_DONE READY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
time = 0x6807c94e 0x32c5bc76
eid = 0x12
Apr 22 2025 21:52:30.851819638 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x19675c744f600401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x19675c9e72e
vdev_delta_ts = 0x233c0
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x11
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3080b1 [DONT_AGGREGATE SCRUB SCAN_THREAD CANFAIL IO_RETRY DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x12282
zio_timestamp = 0x19675c7b36e
zio_delta = 0x233c0
zio_priority = 0x4 [SCRUB]
zio_offset = 0x14d08a5a00
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
time = 0x6807c94e 0x32c5bc76
eid = 0x13
Apr 22 2025 21:52:30.851819638 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x19675c744f600401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x19675c731d2
vdev_delta_ts = 0x11ddc
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0xf
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x11c42
zio_timestamp = 0x19675c613f6
zio_delta = 0x11ddc
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43e2b0e400
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
time = 0x6807c94e 0x32c5bc76
eid = 0x14
Apr 22 2025 21:52:30.895153419 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x19678508f1f03401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1967850751b
vdev_delta_ts = 0x14bb8
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x12
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x149ba
zio_timestamp = 0x196784f2963
zio_delta = 0x14bb8
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5f000
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x403
bad_range_clears = 0x41e
time = 0x6807c94e 0x355af50b
eid = 0x15
Apr 22 2025 21:52:30.908486891 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x196791da1af01c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x196791d848b
vdev_delta_ts = 0x1429e
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x14
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x140c8
zio_timestamp = 0x196791c41ed
zio_delta = 0x1429e
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5e800
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x402
bad_range_clears = 0x3ac
time = 0x6807c94e 0x362668eb
eid = 0x16
Apr 22 2025 21:52:31.165156201 ereport.fs.zfs.data
class = "ereport.fs.zfs.data"
ena = 0x196888d180f01c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_err = 0x34
zio_flags = 0x30000b0 [SCRUB SCAN_THREAD CANFAIL RAW_COMPRESS RAW_ENCRYPT]
zio_stage = 0x4000000 [DONE]
zio_pipeline = 0x4100c02 [READ_BP_INIT DDT_READ_START DDT_READ_DONE READY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
time = 0x6807c94f 0x9d81569
eid = 0x17
Apr 22 2025 21:52:31.165156201 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x196888d180f01c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x196888ff269
vdev_delta_ts = 0x24734
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x18
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3080b1 [DONT_AGGREGATE SCRUB SCAN_THREAD CANFAIL IO_RETRY DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x13312
zio_timestamp = 0x196888dab35
zio_delta = 0x24734
zio_priority = 0x4 [SCRUB]
zio_offset = 0x10fcad0800
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
time = 0x6807c94f 0x9d81569
eid = 0x18
Apr 22 2025 21:52:31.165156201 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x196888d180f01c01
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x196888cf505
vdev_delta_ts = 0x11de6
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x16
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x11c6a
zio_timestamp = 0x196888bd71f
zio_delta = 0x11de6
zio_priority = 0x4 [SCRUB]
zio_offset = 0x39f486da00
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
time = 0x6807c94f 0x9d81569
eid = 0x19
Apr 22 2025 21:52:31.215156716 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1968b7e34d103401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1968b7e038f
vdev_delta_ts = 0x13b96
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x19
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x138da
zio_timestamp = 0x1968b7cc7f9
zio_delta = 0x13b96
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5ec00
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x3ff
bad_range_clears = 0x41c
time = 0x6807c94f 0xcd307ec
eid = 0x1a
Apr 22 2025 21:52:31.365158258 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x196946caa3a03801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x196946c85e6
vdev_delta_ts = 0xcf62
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x1b
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0xcd8c
zio_timestamp = 0x196946bb684
zio_delta = 0xcf62
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5ea00
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x3ff
bad_range_clears = 0x40a
time = 0x6807c94f 0x15c3df72
eid = 0x1b
Apr 22 2025 21:52:31.451825817 ereport.fs.zfs.data
class = "ereport.fs.zfs.data"
ena = 0x19699a31b9d03801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_err = 0x34
zio_flags = 0x30000b0 [SCRUB SCAN_THREAD CANFAIL RAW_COMPRESS RAW_ENCRYPT]
zio_stage = 0x4000000 [DONE]
zio_pipeline = 0x4100c02 [READ_BP_INIT DDT_READ_START DDT_READ_DONE READY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
time = 0x6807c94f 0x1aee5099
eid = 0x1c
Apr 22 2025 21:52:31.451825817 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x19699a31b9d03801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x19699a5e680
vdev_delta_ts = 0x23191
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x1f
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3080b1 [DONT_AGGREGATE SCRUB SCAN_THREAD CANFAIL IO_RETRY DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x11a30
zio_timestamp = 0x19699a3b4ef
zio_delta = 0x23191
zio_priority = 0x4 [SCRUB]
zio_offset = 0x14d08a5200
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
time = 0x6807c94f 0x1aee5099
eid = 0x1d
Apr 22 2025 21:52:31.451825817 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x19699a31b9d03801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x19699a2f889
vdev_delta_ts = 0x11d1e
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x1d
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x11b66
zio_timestamp = 0x19699a1db6b
zio_delta = 0x11d1e
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43e2b0dc00
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
time = 0x6807c94f 0x1aee5099
eid = 0x1e
Apr 22 2025 21:52:31.458492551 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1969a003a0d03801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1969a00222f
vdev_delta_ts = 0xcad0
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x20
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0xc940
zio_timestamp = 0x19699ff575f
zio_delta = 0xcad0
zio_priority = 0x4 [SCRUB]
zio_offset = 0x39703f1600
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x41a
bad_range_clears = 0x3ba
time = 0x6807c94f 0x1b540a87
eid = 0x1f
Apr 22 2025 21:52:31.501826331 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1969c8e2ad703801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1969c8e15fb
vdev_delta_ts = 0xcec2
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x22
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0xcd00
zio_timestamp = 0x1969c8d4739
zio_delta = 0xcec2
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5f200
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x424
bad_range_clears = 0x406
time = 0x6807c94f 0x1de9431b
eid = 0x20
Apr 22 2025 21:52:31.518493170 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1969d9b868903801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1969d9b7397
vdev_delta_ts = 0xc9b8
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x24
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0xc846
zio_timestamp = 0x1969d9aa9df
zio_delta = 0xc9b8
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5f600
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x3ee
bad_range_clears = 0x40b
time = 0x6807c94f 0x1ee793f2
eid = 0x21
Apr 22 2025 21:52:31.531826639 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1969e55d9df03401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1969e55c55d
vdev_delta_ts = 0xcb2a
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x26
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0xc990
zio_timestamp = 0x1969e54fa33
zio_delta = 0xcb2a
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5fc00
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x418
bad_range_clears = 0x3ed
time = 0x6807c94f 0x1fb307cf
eid = 0x22
Apr 22 2025 21:52:31.728495328 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x196aa26a29b01401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x196aa2679fb
vdev_delta_ts = 0xce37
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x28
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0xcc61
zio_timestamp = 0x196aa25abc4
zio_delta = 0xce37
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5fe00
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x3f9
bad_range_clears = 0x40e
time = 0x6807c94f 0x2b6bf4e0
eid = 0x23
Apr 22 2025 21:52:31.795162680 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x196ae1c5f5001401
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x196ae1c4844
vdev_delta_ts = 0xc99a
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x2a
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0xc80a
zio_timestamp = 0x196ae1b7eaa
zio_delta = 0xc99a
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5ee00
zio_size = 0x200
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "gang_header"
bad_ranges = 0x0 0x200
bad_ranges_min_gap = 0x8
bad_range_sets = 0x429
bad_range_clears = 0x3ef
time = 0x6807c94f 0x2f653838
eid = 0x24
Apr 22 2025 21:54:08.735849892 ereport.fs.zfs.data
class = "ereport.fs.zfs.data"
ena = 0x1ad30f2c15501001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_err = 0x34
zio_flags = 0x30000b0 [SCRUB SCAN_THREAD CANFAIL RAW_COMPRESS RAW_ENCRYPT]
zio_stage = 0x4000000 [DONE]
zio_pipeline = 0x4100c02 [READ_BP_INIT DDT_READ_START DDT_READ_DONE READY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
time = 0x6807c9b0 0x2bdc2da4
eid = 0x25
Apr 22 2025 21:54:08.735849892 ereport.fs.zfs.checksum
class = "ereport.fs.zfs.checksum"
ena = 0x1ad30f2c15501001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x21d443bb0bc90fb3
vdev = 0xee662c23d4dc91cf
(end detector)
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
vdev_guid = 0xee662c23d4dc91cf
vdev_type = "disk"
vdev_path = "/dev/disk/by-id/nvme-SAMSUNG_MZVL21T0HCLR-00B00_S676NXMT800285-part6"
vdev_ashift = 0x9
vdev_complete_ts = 0x1ad30f1e7f3
vdev_delta_ts = 0xbff388
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x2c
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x21d443bb0bc90fb3
parent_type = "root"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x34
zio_flags = 0x3000b0 [SCRUB SCAN_THREAD CANFAIL DONT_QUEUE DONT_PROPAGATE]
zio_stage = 0x1000000 [CHECKSUM_VERIFY]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x4c055
zio_timestamp = 0x1ad3022694f
zio_delta = 0xc9a07e
zio_priority = 0x4 [SCRUB]
zio_offset = 0x43ddc5e800
zio_size = 0xaba00
zio_objset = 0x0
zio_object = 0x0
zio_level = 0x0
zio_blkid = 0x0
cksum_algorithm = "sha256"
time = 0x6807c9b0 0x2bdc2da4
eid = 0x26
Apr 22 2025 21:54:24.879246068 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
history_hostname = "alpha"
history_internal_str = "errors=2"
history_internal_name = "scan done"
history_txg = 0x1db78a
history_time = 0x6807c9c0
time = 0x6807c9c0 0x34683af4
eid = 0x27
Apr 22 2025 21:54:24.879246068 sysevent.fs.zfs.scrub_finish
version = 0x0
class = "sysevent.fs.zfs.scrub_finish"
pool = "sz"
pool_guid = 0x21d443bb0bc90fb3
pool_state = 0x0
pool_context = 0x0
time = 0x6807c9c0 0x34683af4
eid = 0x28
You are not alone with this problem - see https://github.com/openzfs/zfs/issues/13483 Doesn't matter if it needs to be imported or is already running (at least for us)
Our setup is similar to yours - zstd compression, dedup, encryption, and running "low" free space (definitely > 50GB but still it seems ZFS required more)
We haven't found a true solution to this problem yet either. We are just in the process of rebuilding a new ZFS volume and copying the data over from the old one since nothing we do seems to fix this error.
I think running low on free space is the clue here to reproduce this.
yes, I think you are right!
I can report that, under very similar conditions, I also have... unfortunately, reproduced this error! :P
I've reproduced this error as well. I had recently added some disks to the pool after my pool had reached 100% capacity (which I had already cleared by removing some data). I also make heavy use of dedup, specifically on fully duplicate files that will guarantee deduplication.
A lot of my data is volatile and can be regenerated later. I cleared out around 35T of space (total pool size is 42T) and left it scrubbing on an Ubuntu Live USB for a while. The Ubuntu module mounted the filesystem successfully and after a couple hours seems to have cleared something out. It fixed 2 checksum errors and re-scrubbing doesn't seem to fix any more errors.
I can now boot into my normal system (Gentoo) without it freezing on boot. It will now freeze after several hours of operation instead. It still throws a panic during boot, but skips over it and appears to function normally afterwards (I'm not sure I understand what's happening there).
I have it running right now, this is the dmesg stack trace from the latest boot:
[ 127.397195] PANIC: zfs: adding existent segment to range tree (offset=30c86a12000 size=1000)
[ 127.397945] Showing stack for process 1673
[ 127.398557] CPU: 6 UID: 0 PID: 1673 Comm: txg_sync Tainted: P OE 6.12.31-gentoo-x86_64 #1
[ 127.398560] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 127.398561] Hardware name: Micro-Star International Co., Ltd. MS-7C35/MEG X570 UNIFY (MS-7C35), BIOS A.J0 03/29/2024
[ 127.398562] Call Trace:
[ 127.398564] <TASK>
[ 127.398567] dump_stack_lvl+0x5a/0x80
[ 127.398575] spl_panic+0x10a/0x330 [spl]
[ 127.398580] spl_panic+0x180/0x330 [spl]
[ 127.398584] ? brt_unload+0x1fd9/0x2f80 [zfs]
[ 127.398641] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.398643] ? brt_unload+0x43e/0x2f80 [zfs]
[ 127.398690] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.398691] ? mutex_lock+0x12/0x30
[ 127.398693] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.398694] ? dbuf_find+0x241/0x2b0 [zfs]
[ 127.398738] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.398740] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.398741] ? arc_write+0x4050/0x4840 [zfs]
[ 127.398787] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.398788] ? arc_buf_access+0x100/0x190 [zfs]
[ 127.398831] zfs_panic_recover+0x71/0x90 [zfs]
[ 127.398890] zfs_range_tree_adjust_fill+0x85e/0x23f0 [zfs]
[ 127.398951] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.398952] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.398953] ? dbuf_read+0x59a/0x9d0 [zfs]
[ 127.399007] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.399009] zfs_range_tree_remove_xor_add_segment+0x547/0x660 [zfs]
[ 127.399067] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.399069] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.399070] ? space_map_write+0x42f/0x930 [zfs]
[ 127.399125] zfs_range_tree_remove_xor_add+0x79/0x180 [zfs]
[ 127.399180] metaslab_sync+0x279/0xa00 [zfs]
[ 127.399236] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.399238] vdev_sync+0x72/0xef0 [zfs]
[ 127.399290] ? srso_alias_return_thunk+0x5/0xfbef5
[ 127.399292] ? srso_alias_return_thunk+0x5/0xfbeff
[ 127.399294] spa_sync+0x62e/0x1960 [zfs]
[ 127.399350] ? spa_txg_history_init_io+0x17f/0x1a0 [zfs]
[ 127.399403] txg_wait_open+0x30a/0x4b0 [zfs]
[ 127.399451] ? txg_wait_open+0x100/0x4b0 [zfs]
[ 127.399495] ? spl_taskq_fini+0xb0/0x120 [spl]
[ 127.399499] spl_taskq_fini+0x10d/0x120 [spl]
[ 127.399503] kthread+0xd2/0x100
[ 127.399505] ? __pfx_kthread+0x10/0x10
[ 127.399507] ret_from_fork+0x34/0x50
[ 127.399510] ? __pfx_kthread+0x10/0x10
[ 127.399512] ret_from_fork_asm+0x1a/0x30
[ 127.399516] </TASK>
The Ubuntu image is using zfs-kmod 2.2.6. My Gentoo machine is using 2.3.3.
I've downgraded my machine to zfs-kmod 2.2.7 and this issue has gone away for me.