zfs
zfs copied to clipboard
Published
20 hours ago •
openzfs
openzfs
Panic while receiving and encrypting dataset
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Ubuntu |
| Distribution Version | 22.04 |
| Kernel Version | 5.15.0-30-generic |
| Architecture | x86_64 |
| OpenZFS Version | zfs-2.1.2-1ubuntu3 |
Describe the problem you're observing
While receiving an unencrypted dataset that was then compressed and encrypted, the system paniced.
Describe how to reproduce the problem
I don't have a surefire way to reproduce. But it did happen twice in a matter of minutes. Send options -L -p, receive options -x encryption -x compression.
Include any warning/errors/backtraces from the system logs
[18664.867056] ------------[ cut here ]------------
[18664.867059] virt_to_cache: Object is not a Slab page!
[18664.867077] WARNING: CPU: 61 PID: 2590174 at mm/slab.h:413 kmem_cache_free+0x22d/0x290
[18664.867085] Modules linked in: tls unix_diag cdc_ether usbnet joydev mii br_netfilter xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat bpfilter overlay binfmt_misc veth nvme_fabrics vhost_vsock vmw_vsock_virtio_transport_common vsock vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_counter nf_tables nfnetlink bridge stp llc intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm ipmi_ssif rapl efi_pstore zfs(PO) zunicode(PO) zzstd(O) zlua(O) nls_iso8859_1 zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) input_leds ccp k10temp ptdma acpi_ipmi ipmi_si mac_hid dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua jc42 ipmi_devintf ipmi_msghandler msr ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath
[18664.867145] linear raid1 hid_generic ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm usbhid drm_kms_helper hid syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd cec rc_core nvme ahci xhci_pci drm tg3 i2c_piix4 xhci_pci_renesas nvme_core libahci
[18664.867166] CPU: 61 PID: 2590174 Comm: receive_writer Tainted: P O 5.15.0-30-generic #31-Ubuntu
[18664.867168] Hardware name: TYAN B8036T65AV12E16HR/S8036GM2NE, BIOS V1.00.B20 10/31/2019
[18664.867170] RIP: 0010:kmem_cache_free+0x22d/0x290
[18664.867173] Code: 01 0f 87 00 6e 98 00 83 e3 01 0f 85 64 ff ff ff 48 c7 c6 60 11 04 b0 48 c7 c7 88 a8 3f b0 c6 05 19 4b eb 01 01 e8 83 8c 96 00 <0f> 0b e9 43 ff ff ff ba 01 00 00 00 48 8d 75 c8 4c 89 ff e8 4b 92
[18664.867174] RSP: 0018:ffffbb4a5aedf990 EFLAGS: 00010286
[18664.867176] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000027
[18664.867177] RDX: ffff9c9f4bd60988 RSI: 0000000000000001 RDI: ffff9c9f4bd60980
[18664.867178] RBP: ffffbb4a5aedf9d8 R08: 0000000000000003 R09: ffffffffffe386f0
[18664.867179] R10: ffffffffffffffff R11: 0000000000000001 R12: ffff9c508d650000
[18664.867180] R13: ffff9c510d650000 R14: ffff9c508d650000 R15: 0000000000004000
[18664.867181] FS: 0000000000000000(0000) GS:ffff9c9f4bd40000(0000) knlGS:0000000000000000
[18664.867183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18664.867184] CR2: 00007fb775966010 CR3: 0000001cfd9a8000 CR4: 0000000000350ee0
[18664.867185] Call Trace:
[18664.867187] <TASK>
[18664.867190] spl_kmem_cache_free+0xc9/0x130 [spl]
[18664.867201] zio_buf_free+0x33/0x80 [zfs]
[18664.867388] abd_return_buf+0x2d/0x40 [zfs]
[18664.867470] zio_crypt_copy_dnode_bonus+0x11d/0x170 [zfs]
[18664.867566] arc_buf_untransform_in_place.constprop.0+0x2a/0x40 [zfs]
[18664.867648] arc_buf_fill+0x2c0/0x4d0 [zfs]
[18664.867730] arc_untransform+0x23/0x80 [zfs]
[18664.867812] dbuf_read_verify_dnode_crypt+0xe7/0x180 [zfs]
[18664.867892] dbuf_read_bonus+0x23/0x110 [zfs]
[18664.867971] dbuf_read_impl.constprop.0+0x303/0x490 [zfs]
[18664.868050] ? dbuf_rele_and_unlock+0x12c/0x530 [zfs]
[18664.868129] ? wmsum_add+0xe/0x10 [zfs]
[18664.868208] ? arc_buf_access+0x229/0x240 [zfs]
[18664.868286] ? __cond_resched+0x1a/0x50
[18664.868291] ? do_raw_spin_unlock+0x9/0x10 [zfs]
[18664.868369] ? __raw_spin_unlock+0x9/0x10 [zfs]
[18664.868447] ? __cond_resched+0x1a/0x50
[18664.868450] ? down_read+0x13/0x90
[18664.868452] dbuf_read+0x1b6/0x5a0 [zfs]
[18664.868531] ? __cond_resched+0x1a/0x50
[18664.868533] ? do_raw_spin_unlock+0x9/0x10 [zfs]
[18664.868616] dmu_bonus_hold_by_dnode+0x8d/0x180 [zfs]
[18664.868696] receive_object+0x48e/0xa60 [zfs]
[18664.868777] ? dnode_rele_and_unlock+0x69/0xe0 [zfs]
[18664.868858] ? spl_kmem_free_impl+0x25/0x30 [spl]
[18664.868866] ? __list_del_entry+0x1d/0x30 [zfs]
[18664.868939] ? list_del+0xd/0x30 [zfs]
[18664.869012] receive_process_record+0xbf/0x230 [zfs]
[18664.869087] receive_writer_thread+0xb3/0x180 [zfs]
[18664.869161] ? receive_process_record+0x230/0x230 [zfs]
[18664.869236] thread_generic_wrapper+0x64/0x70 [spl]
[18664.869243] ? __thread_exit+0x20/0x20 [spl]
[18664.869250] kthread+0x12a/0x150
[18664.869254] ? set_kthread_struct+0x50/0x50
[18664.869256] ret_from_fork+0x22/0x30
[18664.869260] </TASK>
[18664.869261] ---[ end trace 50b300a59d683227 ]---
[19408.896773] BUG: unable to handle page fault for address: 0000000100000000
[19408.896807] #PF: supervisor write access in kernel mode
[19408.896824] #PF: error_code(0x0002) - not-present page
[19408.896840] PGD 0 P4D 0
[19408.896854] Oops: 0002 [#1] SMP NOPTI
[19408.896873] CPU: 98 PID: 1011197 Comm: receive_writer Tainted: P W O 5.15.0-30-generic #31-Ubuntu
[19408.896906] Hardware name: TYAN B8036T65AV12E16HR/S8036GM2NE, BIOS V1.00.B20 10/31/2019
[19408.896928] RIP: 0010:__memcpy+0x12/0x20
[19408.896945] Code: cc cc cc cc cc cc cc cc 48 8b 05 a1 23 8e 01 c3 cc cc cc cc cc cc cc cc 0f 1f 44 00 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 f3 a4
[19408.896995] RSP: 0018:ffffbb491f1bb878 EFLAGS: 00010246
[19408.897011] RAX: 0000000100000000 RBX: ffffbb491f1bb928 RCX: 0000000000000800
[19408.897032] RDX: 0000000000000000 RSI: ffff9c50dc884000 RDI: 0000000100000000
[19408.897053] RBP: ffffbb491f1bb890 R08: ffff9c50dc884000 R09: 0000000000000000
[19408.897072] R10: ffff9c51bb1e7500 R11: 000000000000000c R12: 0000000000004000
[19408.897093] R13: 0000000000000000 R14: ffffbb491f1bb928 R15: 0000000000000000
[19408.897114] FS: 0000000000000000(0000) GS:ffff9c9f4c680000(0000) knlGS:0000000000000000
[19408.897137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[19408.897153] CR2: 0000000100000000 CR3: 00000028c25c8000 CR4: 0000000000350ee0
[19408.897174] Call Trace:
[19408.897184] <TASK>
[19408.897194] ? abd_copy_to_buf_off_cb+0x23/0x30 [zfs]
[19408.897282] abd_iterate_func+0xbb/0x100 [zfs]
[19408.897360] ? abd_advance_abd_iter+0x80/0x80 [zfs]
[19408.897440] abd_copy_to_buf_off+0x3c/0x60 [zfs]
[19408.897518] abd_copy_to_buf+0xe/0x10 [zfs]
[19408.897596] abd_borrow_buf_copy+0x42/0x50 [zfs]
[19408.897675] zio_crypt_copy_dnode_bonus+0x33/0x170 [zfs]
[19408.897783] ? do_raw_spin_unlock+0x9/0x10 [zfs]
[19408.897862] arc_buf_untransform_in_place.constprop.0+0x2a/0x40 [zfs]
[19408.897947] arc_buf_fill+0x2c0/0x4d0 [zfs]
[19408.898026] arc_untransform+0x23/0x80 [zfs]
[19408.898104] dbuf_read_verify_dnode_crypt+0xe7/0x180 [zfs]
[19408.898189] dbuf_read_impl.constprop.0+0xf0/0x490 [zfs]
[19408.898280] ? do_raw_spin_unlock+0x9/0x10 [zfs]
[19408.898360] ? __cond_resched+0x1a/0x50
[19408.898374] ? down_read+0x13/0x90
[19408.898386] dbuf_read+0x1b6/0x5a0 [zfs]
[19408.898466] dmu_tx_check_ioerr+0x6e/0xd0 [zfs]
[19408.898551] dmu_tx_hold_free_impl+0x151/0x290 [zfs]
[19408.898634] dmu_tx_hold_free+0x40/0x50 [zfs]
[19408.898717] dmu_free_long_range_impl+0x120/0x350 [zfs]
[19408.898803] dmu_free_long_range+0x74/0xc0 [zfs]
[19408.898886] dmu_free_long_object+0x27/0xc0 [zfs]
[19408.898968] receive_freeobjects+0x85/0x100 [zfs]
[19408.899052] receive_process_record+0x159/0x230 [zfs]
[19408.899136] receive_writer_thread+0xb3/0x180 [zfs]
[19408.899219] ? receive_process_record+0x230/0x230 [zfs]
[19408.899273] thread_generic_wrapper+0x64/0x70 [spl]
[19408.899273] ? __thread_exit+0x20/0x20 [spl]
[19408.899273] kthread+0x12a/0x150
[19408.899273] ? set_kthread_struct+0x50/0x50
[19408.899273] ret_from_fork+0x22/0x30
[19408.899273] </TASK>
[19408.899273] Modules linked in: tls unix_diag cdc_ether usbnet joydev mii br_netfilter xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat bpfilter overlay binfmt_misc veth nvme_fabrics vhost_vsock vmw_vsock_virtio_transport_common vsock vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_counter nf_tables nfnetlink bridge stp llc intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm ipmi_ssif rapl efi_pstore zfs(PO) zunicode(PO) zzstd(O) zlua(O) nls_iso8859_1 zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) input_leds ccp k10temp ptdma acpi_ipmi ipmi_si mac_hid dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua jc42 ipmi_devintf ipmi_msghandler msr ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath
[19408.899273] linear raid1 hid_generic ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm usbhid drm_kms_helper hid syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd cec rc_core nvme ahci xhci_pci drm tg3 i2c_piix4 xhci_pci_renesas nvme_core libahci
[19408.903187] CR2: 0000000100000000
[19408.903574] ---[ end trace 50b300a59d683228 ]---
[19408.983169] RIP: 0010:__memcpy+0x12/0x20
[19409.013186] Code: cc cc cc cc cc cc cc cc 48 8b 05 a1 23 8e 01 c3 cc cc cc cc cc cc cc cc 0f 1f 44 00 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 f3 a4
[19409.014940] RSP: 0018:ffffbb491f1bb878 EFLAGS: 00010246
[19409.016205] RAX: 0000000100000000 RBX: ffffbb491f1bb928 RCX: 0000000000000800
[19409.017341] RDX: 0000000000000000 RSI: ffff9c50dc884000 RDI: 0000000100000000
[19409.018089] RBP: ffffbb491f1bb890 R08: ffff9c50dc884000 R09: 0000000000000000
[19409.019396] R10: ffff9c51bb1e7500 R11: 000000000000000c R12: 0000000000004000
[19409.019728] R13: 0000000000000000 R14: ffffbb491f1bb928 R15: 0000000000000000
[19409.020799] FS: 0000000000000000(0000) GS:ffff9c9f4c680000(0000) knlGS:0000000000000000
[19409.021727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[19409.022719] CR2: 0000000100000000 CR3: 00000028c25c8000 CR4: 0000000000350ee0
The pool was hard-locked after that but I was able to do a regular reboot. Shortly after I got the next panic while receiving another dataset:
[ 157.268524] ------------[ cut here ]------------
[ 157.268528] virt_to_cache: Object is not a Slab page!
[ 157.268536] WARNING: CPU: 121 PID: 198913 at mm/slab.h:413 kmem_cache_free+0x22d/0x290
[ 157.268549] Modules linked in: br_netfilter unix_diag xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat bpfilter overlay binfmt_misc veth vhost_vsock vmw_vsock_virtio_transport_com mon vsock nvme_fabrics vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf _defrag_ipv4 nft_counter nf_tables nfnetlink bridge stp llc intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd ipmi_ssif kvm rapl efi_pstore zfs(PO) zunicode(PO) zzstd(O) zlua(O) nls_iso8859_1 zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) joydev input_leds ccp k10temp ptdma acpi_ipmi ipmi_si mac_hid sch_fq_codel jc42 ipmi_devintf dm_multipath ipmi_msghandler scsi_dh _rdac scsi_dh_emc scsi_dh_alua msr ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc3 2c raid0 multipath linear raid1 hid_generic
[ 157.268647] ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm cdc_ether drm_kms_helper usbnet mii syscopyarea sysfillrect crct10dif_pclmul sysimgblt crc32_pclmul fb_sys_fops usbhid ghash _clmulni_intel hid aesni_intel crypto_simd cryptd cec rc_core nvme ahci xhci_pci drm i2c_piix4 tg3 xhci_pci_renesas nvme_core libahci
[ 157.268683] CPU: 121 PID: 198913 Comm: receive_writer Tainted: P O 5.15.0-30-generic #31-Ubuntu
[ 157.268688] Hardware name: TYAN B8036T65AV12E16HR/S8036GM2NE, BIOS V1.00.B20 10/31/2019
[ 157.268691] RIP: 0010:kmem_cache_free+0x22d/0x290
[ 157.268697] Code: 01 0f 87 00 6e 98 00 83 e3 01 0f 85 64 ff ff ff 48 c7 c6 60 11 44 bd 48 c7 c7 88 a8 7f bd c6 05 19 4b eb 01 01 e8 83 8c 96 00 <0f> 0b e9 43 ff ff ff ba 01 00 00 00 48 8d 7 5 c8 4c 89 ff e8 4b 92
[ 157.268700] RSP: 0018:ffffa564732138b8 EFLAGS: 00010286
[ 157.268703] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000027
[ 157.268706] RDX: ffff96454cc60988 RSI: 0000000000000001 RDI: ffff96454cc60980
[ 157.268708] RBP: ffffa56473213900 R08: 0000000000000003 R09: ffffffffffe382f8
[ 157.268710] R10: ffffffffffffffff R11: 0000000000000001 R12: ffff95f423424000
[ 157.268712] R13: ffff95f4a3424000 R14: ffff95f423424000 R15: 0000000000004000
[ 157.268715] FS: 0000000000000000(0000) GS:ffff96454cc40000(0000) knlGS:0000000000000000
[ 157.268718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 157.268720] CR2: 00001a49cdb42010 CR3: 00000028b04b4000 CR4: 0000000000350ee0
[ 157.268723] Call Trace:
[ 157.268726] <TASK>
[ 157.268732] spl_kmem_cache_free+0xc9/0x130 [spl]
[ 157.268751] zio_buf_free+0x33/0x80 [zfs]
[ 157.268972] abd_return_buf+0x2d/0x40 [zfs]
[ 157.269121] zio_crypt_copy_dnode_bonus+0x11d/0x170 [zfs]
[ 157.269319] arc_buf_untransform_in_place.constprop.0+0x2a/0x40 [zfs]
[ 157.269470] arc_buf_fill+0x2c0/0x4d0 [zfs]
[ 157.269620] ? psi_task_switch+0xc6/0x220
[ 157.269626] arc_untransform+0x23/0x80 [zfs]
[ 157.269776] dbuf_read_verify_dnode_crypt+0xe7/0x180 [zfs]
[ 157.269931] dbuf_read_impl.constprop.0+0xf0/0x490 [zfs]
[ 157.270085] ? do_raw_spin_unlock+0x9/0x10 [zfs]
[ 157.270235] ? __cond_resched+0x1a/0x50
[ 157.270241] ? down_read+0x13/0x90
[ 157.270244] dbuf_read+0x1b6/0x5a0 [zfs]
[ 157.270398] dmu_tx_check_ioerr+0x6e/0xd0 [zfs]
[ 157.270561] dmu_tx_hold_free_impl+0x151/0x290 [zfs]
[ 157.270729] dmu_tx_hold_free+0x40/0x50 [zfs]
[ 157.270888] dmu_free_long_range_impl+0x120/0x350 [zfs]
[ 157.271048] dmu_free_long_range+0x74/0xc0 [zfs]
[ 157.271204] dmu_free_long_object+0x27/0xc0 [zfs]
[ 157.271359] receive_freeobjects+0x85/0x100 [zfs]
[ 157.271518] receive_process_record+0x159/0x230 [zfs]
[ 157.271675] receive_writer_thread+0xb3/0x180 [zfs]
[ 157.271832] ? receive_process_record+0x230/0x230 [zfs]
[ 157.271908] thread_generic_wrapper+0x64/0x70 [spl]
[ 157.271916] ? __thread_exit+0x20/0x20 [spl]
[ 157.271923] kthread+0x12a/0x150
[ 157.271926] ? set_kthread_struct+0x50/0x50
[ 157.271928] ret_from_fork+0x22/0x30
[ 157.271932] </TASK>
[ 157.271933] ---[ end trace 0bd55b9717fb7a97 ]---
This time the pool was not frozen, but I rebooted just in case.
Receiving Pool stats:
pool: tank2
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
tank2 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme-Micron_9200_MTFDHAL3T8TCT_17451CC81F6C ONLINE 0 0 0
nvme-Micron_9200_MTFDHAL3T8TCT_17451CC81EBE ONLINE 0 0 0
errors: No known data errors
NAME PROPERTY VALUE SOURCE
tank2 size 3.48T -
tank2 capacity 18% -
tank2 altroot - default
tank2 health ONLINE -
tank2 guid 4747071815621839097 -
tank2 version - default
tank2 bootfs - default
tank2 delegation on default
tank2 autoreplace off default
tank2 cachefile - default
tank2 failmode wait default
tank2 listsnapshots off default
tank2 autoexpand off default
tank2 dedupratio 1.00x -
tank2 free 2.84T -
tank2 allocated 664G -
tank2 readonly off -
tank2 ashift 12 local
tank2 comment - default
tank2 expandsize - -
tank2 freeing 0 -
tank2 fragmentation 0% -
tank2 leaked 0 -
tank2 multihost off default
tank2 checkpoint - -
tank2 load_guid 13208566083222792801 -
tank2 autotrim on local
tank2 compatibility off default
tank2 feature@async_destroy enabled local
tank2 feature@empty_bpobj active local
tank2 feature@lz4_compress active local
tank2 feature@multi_vdev_crash_dump enabled local
tank2 feature@spacemap_histogram active local
tank2 feature@enabled_txg active local
tank2 feature@hole_birth active local
tank2 feature@extensible_dataset active local
tank2 feature@embedded_data active local
tank2 feature@bookmarks enabled local
tank2 feature@filesystem_limits enabled local
tank2 feature@large_blocks enabled local
tank2 feature@large_dnode active local
tank2 feature@sha512 active local
tank2 feature@skein enabled local
tank2 feature@edonr enabled local
tank2 feature@userobj_accounting active local
tank2 feature@encryption active local
tank2 feature@project_quota active local
tank2 feature@device_removal enabled local
tank2 feature@obsolete_counts enabled local
tank2 feature@zpool_checkpoint enabled local
tank2 feature@spacemap_v2 active local
tank2 feature@allocation_classes enabled local
tank2 feature@resilver_defer enabled local
tank2 feature@bookmark_v2 enabled local
tank2 feature@redaction_bookmarks enabled local
tank2 feature@redacted_datasets enabled local
tank2 feature@bookmark_written enabled local
tank2 feature@log_spacemap active local
tank2 feature@livelist active local
tank2 feature@device_rebuild enabled local
tank2 feature@zstd_compress active local
tank2 feature@draid enabled local
NAME PROPERTY VALUE SOURCE
tank2 type filesystem -
tank2 creation Sat May 21 17:03 2022 -
tank2 used 682G -
tank2 available 2.71T -
tank2 referenced 200K -
tank2 compressratio 1.73x -
tank2 mounted yes -
tank2 quota none default
tank2 reservation none default
tank2 recordsize 128K default
tank2 mountpoint /tank2 default
tank2 sharenfs off default
tank2 checksum sha512 local
tank2 compression zstd-1 local
tank2 atime off local
tank2 devices on default
tank2 exec on default
tank2 setuid on default
tank2 readonly off default
tank2 zoned off default
tank2 snapdir hidden default
tank2 aclmode passthrough local
tank2 aclinherit passthrough local
tank2 createtxg 1 -
tank2 canmount on default
tank2 xattr sa local
tank2 copies 1 default
tank2 version 5 -
tank2 utf8only on -
tank2 normalization formD -
tank2 casesensitivity mixed -
tank2 vscan off default
tank2 nbmand off default
tank2 sharesmb off default
tank2 refquota none default
tank2 refreservation none default
tank2 guid 3540662221705091815 -
tank2 primarycache all default
tank2 secondarycache all default
tank2 usedbysnapshots 0B -
tank2 usedbydataset 200K -
tank2 usedbychildren 682G -
tank2 usedbyrefreservation 0B -
tank2 logbias latency default
tank2 objsetid 54 -
tank2 dedup off default
tank2 mlslabel none default
tank2 sync standard default
tank2 dnodesize auto local
tank2 refcompressratio 1.00x -
tank2 written 200K -
tank2 logicalused 1.05T -
tank2 logicalreferenced 69.5K -
tank2 volmode default default
tank2 filesystem_limit none default
tank2 snapshot_limit none default
tank2 filesystem_count none default
tank2 snapshot_count none default
tank2 snapdev hidden default
tank2 acltype posix local
tank2 context none default
tank2 fscontext none default
tank2 defcontext none default
tank2 rootcontext none default
tank2 relatime off default
tank2 redundant_metadata all default
tank2 overlay on default
tank2 encryption aes-256-gcm -
tank2 keylocation prompt local
tank2 keyformat passphrase -
tank2 pbkdf2iters 350000 -
tank2 encryptionroot tank2 -
tank2 keystatus available -
tank2 special_small_blocks 0 default
@rincebrain if this is a duplicate, feel free to close. In the meantime, I had the last stack trace again...
It may be the most duplicated bug in recent memory. I don't think they've been being closed so far, though, since some of them are more ambiguously duplicates than others.
Another one here with 2.1.9-1~bpo11+1.
[3514154.883087] ------------[ cut here ]------------
[3514154.888230] virt_to_cache: Object is not a Slab page!
[3514154.893808] WARNING: CPU: 7 PID: 3387058 at mm/slab.h:620 kmem_cache_free+0x35d/0x410
[3514154.902133] Modules linked in: xt_nat nf_conntrack_netlink xfrm_user binfmt_misc veth tun xt_CHECKSUM xt_MASQUERADE xt_tcpudp nft_compat overlay macvlan bridge stp llc nls_ascii nls_cp437 vfat fat nft_nat nft_masq nft_fib_ipv4 nft_fib nft_chain_nat nf_nat nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_limit intel_rapl_msr intel_rapl_common ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm nft_ct irqbypass nf_conntrack ast nf_defrag_ipv6 ghash_clmulni_intel nf_defrag_ipv4 drm_vram_helper acpi_ipmi aesni_intel drm_ttm_helper cdc_ether iTCO_wdt crypto_simd ttm cryptd intel_pmc_bxt usbnet ipmi_si mei_me intel_cstate iTCO_vendor_support drm_kms_helper joydev pcspkr evdev ipmi_devintf intel_uncore efi_pstore mii i2c_algo_bit mei watchdog sg ipmi_msghandler acpi_pad intel_pmc_core acpi_power_meter button acpi_tad nf_tables crc32c_generic nfnetlink fuse drm configfs ip_tables x_tables autofs4 zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE)
[3514154.902177] zcommon(POE) znvpair(POE) spl(OE) efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear md_mod sr_mod cdrom hid_generic usbhid hid uas usb_storage sd_mod ixgbe xfrm_algo ahci xhci_pci dca libahci nvme mdio_devres xhci_hcd libata nvme_core libphy t10_pi intel_ish_ipc intel_lpss_pci i2c_i801 crc64_rocksoft crc32_pclmul ptp scsi_mod crc64 usbcore intel_lpss crc32c_intel i2c_smbus intel_ishtp pps_core crc_t10dif idma64 mdio usb_common crct10dif_generic scsi_common crct10dif_pclmul crct10dif_common fan video
[3514155.046131] CPU: 7 PID: 3387058 Comm: receive_writer Tainted: P OE 6.0.0-0.deb11.6-amd64 #1 Debian 6.0.12-1~bpo11+1
[3514155.058687] Hardware name: GIGABYTE MX33-BS1-V1/MX33-BS1-V1, BIOS F04a 10/01/2021
[3514155.066642] RIP: 0010:kmem_cache_free+0x35d/0x410
[3514155.071823] Code: fd ff ff 80 3d d0 4f 6f 01 00 0f 85 d7 fe ff ff 48 c7 c6 00 de a3 ba 48 c7 c7 48 30 d6 ba c6 05 b5 4f 6f 01 01 e8 29 c9 67 00 <0f> 0b 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc 48 8b 57 60 48 8b 4d
[3514155.091324] RSP: 0018:ffffac7894ee3a30 EFLAGS: 00010286
[3514155.097026] RAX: 0000000000000000 RBX: ffff93be21f74000 RCX: 0000000000000000
[3514155.104658] RDX: 0000000000000001 RSI: ffffffffbad7c802 RDI: 00000000ffffffff
[3514155.112293] RBP: ffff93baf07a6400 R08: 0000000000000000 R09: 00000001000005a4
[3514155.119896] R10: ffffac7894ee38d0 R11: ffffffffbb4c2a20 R12: ffff93bea1f74000
[3514155.127489] R13: ffffffffc0d838a0 R14: 0000000000000000 R15: ffffffffc0d838c0
[3514155.135082] FS: 0000000000000000(0000) GS:ffff93c9dfbc0000(0000) knlGS:0000000000000000
[3514155.143617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3514155.149814] CR2: 00005595f4b1eac0 CR3: 00000001ac810001 CR4: 0000000000772ee0
[3514155.157405] PKRU: 55555554
[3514155.160553] Call Trace:
[3514155.163446] <TASK>
[3514155.165983] spl_kmem_cache_free+0x12e/0x1d0 [spl]
[3514155.171212] arc_buf_fill+0x3f5/0xce0 [zfs]
[3514155.175867] ? dbuf_dirty+0x498/0x9c0 [zfs]
[3514155.180520] arc_untransform+0x1d/0x80 [zfs]
[3514155.185261] dbuf_read_verify_dnode_crypt+0xf2/0x160 [zfs]
[3514155.191191] dbuf_read_impl.constprop.0+0x52a/0x6d0 [zfs]
[3514155.197027] ? percpu_counter_add_batch+0x58/0xc0
[3514155.202150] ? dbuf_cons+0xa7/0xc0 [zfs]
[3514155.206513] ? spl_kmem_cache_alloc+0x9f/0x7d0 [spl]
[3514155.211887] ? dbuf_rele_and_unlock+0x132/0x690 [zfs]
[3514155.217361] ? preempt_count_add+0x70/0xa0
[3514155.221853] ? preempt_count_add+0x70/0xa0
[3514155.226345] ? _raw_spin_lock+0x13/0x40
[3514155.230570] ? aggsum_add+0x178/0x190 [zfs]
[3514155.235160] dbuf_read+0xe5/0x5f0 [zfs]
[3514155.239402] ? dnode_hold_impl+0x4eb/0xee0 [zfs]
[3514155.244439] ? preempt_count_add+0x70/0xa0
[3514155.248921] dmu_bonus_hold_by_dnode+0x86/0x1a0 [zfs]
[3514155.254376] receive_object+0x410/0xca0 [zfs]
[3514155.259146] ? dnode_rele_and_unlock+0x59/0xf0 [zfs]
[3514155.264508] ? preempt_count_add+0x70/0xa0
[3514155.268976] ? __thread_exit+0x20/0x20 [spl]
[3514155.273616] receive_writer_thread+0x1cc/0xad0 [zfs]
[3514155.278972] ? __slab_free+0xa1/0x2d0
[3514155.283000] ? set_next_task_fair+0x2d/0xd0
[3514155.287533] ? _raw_spin_unlock_irqrestore+0x23/0x40
[3514155.292849] ? receive_process_write_record+0x1a0/0x1a0 [zfs]
[3514155.298972] ? __thread_exit+0x20/0x20 [spl]
[3514155.303669] ? thread_generic_wrapper+0x56/0x70 [spl]
[3514155.309066] thread_generic_wrapper+0x56/0x70 [spl]
[3514155.314273] kthread+0xe5/0x110
[3514155.317761] ? kthread_complete_and_exit+0x20/0x20
[3514155.322885] ret_from_fork+0x1f/0x30
[3514155.326789] </TASK>
[3514155.329331] ---[ end trace 0000000000000000 ]---
