zfs Kernel oops page fault triggered by Docker in arc

Kernel oops page fault triggered by Docker in arc_prune

Open maxpoulin64 opened this issue 1 year ago • 83 comments

trafficstars

System information

Type	Version/Name
Distribution Name	ArchLinux
Distribution Version	Latest
Kernel Version	6.8.9-zen1-1-zen
Architecture	x86_64
OpenZFS Version	2.2.4-2

I'm holding to 6.8.9 specifically to stay within official supported kernel versions.

Describe the problem you're observing

Extracting large container images in Docker causes ZFS to trigger an unhandled page fault, and permanently locks up the filesystem until reboot. Sync will never complete, and normal shutdown also doesn't complete.

Describe how to reproduce the problem

Running this particular container reliably hangs ZFS on my system during extraction, using Docker's ZFS storage driver.

docker run -it --rm -p 8080:8080 --gpus all --name localai quay.io/go-skynet/local-ai:latest-aio-gpu-hipblas

It gets stuck on a line such as this one and never completes, killing the Docker daemon makes it a zombie, IO is completely hosed.

6ddbee975253: Extracting  352.2MB/352.2MB

Include any warning/errors/backtraces from the system logs

[184791.050957] BUG: unable to handle page fault for address: 00000000208db6e0
[184791.050969] #PF: supervisor instruction fetch in kernel mode
[184791.050972] #PF: error_code(0x0010) - not-present page
[184791.050975] PGD 0 P4D 0 
[184791.050981] Oops: 0010 [#1] PREEMPT SMP NOPTI
[184791.050985] CPU: 11 PID: 482 Comm: arc_prune Tainted: P        W  OE      6.8.9-zen1-1-zen #1 b3e4ad3c9dbde87c9fb9d46fb90ca62a28a66a12
[184791.050992] Hardware name: Micro-Star International Co., Ltd. MS-7B09/X399 GAMING PRO CARBON AC (MS-7B09), BIOS 1.B0 08/09/2018
[184791.050995] RIP: 0010:0x208db6e0
[184791.051042] Code: Unable to access opcode bytes at 0x208db6b6.
[184791.051045] RSP: 0018:ffffb417d2293ce0 EFLAGS: 00010246
[184791.051049] RAX: 00000000208db6e0 RBX: ffffb417d2293d94 RCX: 0000000000000000
[184791.051052] RDX: 0000000000000000 RSI: ffffb417d2293d30 RDI: ffff97e1ac586a80
[184791.051056] RBP: 0000000000003ae0 R08: 0000000000006d66 R09: ffff97e4860e2e90
[184791.051059] R10: ffff97e4860e2e80 R11: ffff97e1f96c0000 R12: ffff97e538d00000
[184791.051063] R13: ffff97e48bf9d780 R14: ffff97e4860e2e28 R15: ffff97e1ac586a80
[184791.051066] FS:  0000000000000000(0000) GS:ffff97e46e4c0000(0000) knlGS:0000000000000000
[184791.051070] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[184791.051074] CR2: 00000000208db6e0 CR3: 000000019b3e6000 CR4: 00000000003506f0
[184791.051077] Call Trace:
[184791.051082]  <TASK>
[184791.051085]  ? __die+0x10f/0x120
[184791.051092]  ? page_fault_oops+0x171/0x4e0
[184791.051101]  ? exc_page_fault+0x7f/0x180
[184791.051107]  ? asm_exc_page_fault+0x26/0x30
[184791.051119]  ? zfs_prune+0xb0/0x4e0 [zfs 158ff065068c3ea6e221f98356463834dc655cec]
[184791.051438]  ? zpl_prune_sb+0x36/0x60 [zfs 158ff065068c3ea6e221f98356463834dc655cec]
[184791.051653]  ? arc_prune_task+0x22/0x40 [zfs 158ff065068c3ea6e221f98356463834dc655cec]
[184791.051880]  ? taskq_thread+0x2d4/0x6f0 [spl 44541b25f59ba0491e81482257bd475148318e14]
[184791.051901]  ? srso_return_thunk+0x5/0x5f
[184791.051907]  ? finish_task_switch.isra.0+0x94/0x2f0
[184791.051914]  ? __pfx_default_wake_function+0x10/0x10
[184791.051924]  ? __pfx_taskq_thread+0x10/0x10 [spl 44541b25f59ba0491e81482257bd475148318e14]
[184791.051941]  ? kthread+0xe8/0x120
[184791.051946]  ? __pfx_kthread+0x10/0x10
[184791.051951]  ? ret_from_fork+0x34/0x50
[184791.051955]  ? __pfx_kthread+0x10/0x10
[184791.051960]  ? ret_from_fork_asm+0x1b/0x30
[184791.051969]  </TASK>
[184791.051971] Modules linked in: xt_conntrack nf_conntrack_netlink xfrm_user xfrm_algo ip6table_nat ip6table_filter ip6_tables xt_addrtype br_netfilter overlay rfcomm snd_seq_dummy snd_hrtimer snd_seq wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel bridge stp llc uhid cmac algif_hash algif_skcipher af_alg xt_MASQUERADE bnep xt_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_generic iptable_filter dm_crypt cbc encrypted_keys vfat fat intel_rapl_msr intel_rapl_common btusb snd_hda_codec_realtek btrtl crct10dif_pclmul snd_hda_codec_generic btintel crc32_pclmul iwlmvm snd_hda_codec_hdmi btbcm crc32c_intel snd_usb_audio btmtk polyval_clmulni snd_hda_intel snd_usbmidi_lib polyval_generic mac80211 gf128mul libarc4 snd_intel_dspcfg snd_ump ghash_clmulni_intel snd_intel_sdw_acpi bluetooth snd_rawmidi sha512_ssse3 joydev snd_seq_device snd_hda_codec sha256_ssse3 ecdh_generic iwlwifi mousedev sha1_ssse3 mc
[184791.052084]  razerkbd(OE) crc16 aesni_intel snd_hda_core crypto_simd snd_hwdep cryptd snd_pcm igb cfg80211 rapl ptp snd_timer sp5100_tco pps_core gpio_amdpt snd dca wmi_bmof rfkill pcspkr soundcore gpio_generic mxm_wmi i2c_piix4 k10temp mac_hid kvmfr(OE) sg crypto_user loop nfnetlink ip_tables x_tables hid_steam ff_memless hid_logitech_hidpp hid_logitech_dj hid_generic trusted asn1_encoder tee dm_mod usbhid amdgpu vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd video amdxcp i2c_algo_bit drm_ttm_helper ttm kvm_amd drm_exec gpu_sched drm_suballoc_helper kvm nvme drm_buddy nvme_core drm_display_helper xhci_pci irqbypass cec ccp nvme_auth xhci_pci_renesas wmi zfs(POE) spl(OE) vendor_reset(OE) nct6775 nct6775_core hwmon_vid i2c_dev
[184791.052189] CR2: 00000000208db6e0
[184791.052193] ---[ end trace 0000000000000000 ]---
[184791.052196] RIP: 0010:0x208db6e0
[184791.052216] Code: Unable to access opcode bytes at 0x208db6b6.
[184791.052219] RSP: 0018:ffffb417d2293ce0 EFLAGS: 00010246
[184791.052223] RAX: 00000000208db6e0 RBX: ffffb417d2293d94 RCX: 0000000000000000
[184791.052226] RDX: 0000000000000000 RSI: ffffb417d2293d30 RDI: ffff97e1ac586a80
[184791.052229] RBP: 0000000000003ae0 R08: 0000000000006d66 R09: ffff97e4860e2e90
[184791.052232] R10: ffff97e4860e2e80 R11: ffff97e1f96c0000 R12: ffff97e538d00000
[184791.052235] R13: ffff97e48bf9d780 R14: ffff97e4860e2e28 R15: ffff97e1ac586a80
[184791.052238] FS:  0000000000000000(0000) GS:ffff97e46e4c0000(0000) knlGS:0000000000000000
[184791.052241] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[184791.052244] CR2: 00000000208db6e0 CR3: 000000019b3e6000 CR4: 00000000003506f0
[184791.052248] note: arc_prune[482] exited with irqs disabled

The stack trace is always the same. Disk passes scrub with 0 errors after rebooting.

Jul 06 '24 08:07 maxpoulin64

I've seen this several times on several systems running ZFS 2.2.4 and Linux 6.8. Reverting to Linux 6.6 is stable. We also heavily use Docker with ZFS. Our systems are 24 - 128 thread systems running software build jobs in parallel using Docker containers. They see heavy CPU usage and IO over NVMe.

I have a trace which looks similar to yours (arc_prune), and I also started seeing at the same time another NULL pointer deference in a different stack trace referencing iptables. They both happen together so I'll include that trace here, but I don't know if it is actually related.

An example of arc_prune:

May 09 12:08:04 pingu kernel: BUG: unable to handle page fault for address: 0000000200000000
May 09 12:08:04 pingu kernel: #PF: supervisor instruction fetch in kernel mode
May 09 12:08:04 pingu kernel: #PF: error_code(0x0010) - not-present page
May 09 12:08:04 pingu kernel: PGD 15e5cce067 P4D 15e5cce067 PUD 0 
May 09 12:08:04 pingu kernel: Oops: 0010 [#1] PREEMPT SMP NOPTI
May 09 12:08:04 pingu kernel: CPU: 102 PID: 1366 Comm: arc_prune Tainted: P           O       6.8.9 #1
May 09 12:08:04 pingu kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C60/TRX40 PRO 10G (MS-7C60), BIOS 1.60 05/13/2020
May 09 12:08:04 pingu kernel: RIP: 0010:0x200000000
May 09 12:08:04 pingu kernel: Code: Unable to access opcode bytes at 0x1ffffffd6.
May 09 12:08:04 pingu kernel: RSP: 0018:ffffb59e438cfd08 EFLAGS: 00010246
May 09 12:08:04 pingu kernel: RAX: 0000000200000000 RBX: ffff9c0fab2f0000 RCX: 0000000000000000
May 09 12:08:04 pingu kernel: RDX: 0000000000000001 RSI: ffffb59e438cfd58 RDI: ffff9bfb360f7180
May 09 12:08:04 pingu kernel: RBP: ffffb59e438cfdbc R08: 000000000000fdd9 R09: ffff9c38fe7ba4c0
May 09 12:08:04 pingu kernel: R10: 000000000000050a R11: 0000000000000066 R12: 000000000000fdd9
May 09 12:08:04 pingu kernel: R13: ffff9bfaf9386000 R14: ffff9c0fab2f00f8 R15: ffff9bfb360f7180
May 09 12:08:04 pingu kernel: FS:  0000000000000000(0000) GS:ffff9c38fe780000(0000) knlGS:0000000000000000
May 09 12:08:04 pingu kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 09 12:08:04 pingu kernel: CR2: 0000000200000000 CR3: 000000027fb1a000 CR4: 0000000000350ef0
May 09 12:08:04 pingu kernel: Call Trace:
May 09 12:08:04 pingu kernel:  <TASK>
May 09 12:08:04 pingu kernel:  ? __die_body+0x1b/0x60
May 09 12:08:04 pingu kernel:  ? page_fault_oops+0x15d/0x470
May 09 12:08:04 pingu kernel:  ? __mod_node_page_state+0x82/0xc0
May 09 12:08:04 pingu kernel:  ? exc_page_fault+0x74/0x170
May 09 12:08:04 pingu kernel:  ? asm_exc_page_fault+0x22/0x30
May 09 12:08:04 pingu kernel:  ? _raw_spin_unlock+0x15/0x30
May 09 12:08:04 pingu kernel:  ? zfs_prune+0x9c/0x4a0 [zfs]
May 09 12:08:04 pingu kernel:  ? autoremove_wake_function+0xe/0x30
May 09 12:08:04 pingu kernel:  ? zpl_prune_sb+0x34/0x50 [zfs]
May 09 12:08:04 pingu kernel:  ? arc_prune_task+0x1b/0x30 [zfs]
May 09 12:08:04 pingu kernel:  ? taskq_thread+0x26e/0x470 [spl]
May 09 12:08:04 pingu kernel:  ? wake_up_state+0x10/0x10
May 09 12:08:04 pingu kernel:  ? task_done+0x90/0x90 [spl]
May 09 12:08:04 pingu kernel:  ? kthread+0xee/0x120
May 09 12:08:04 pingu kernel:  ? kthread_complete_and_exit+0x20/0x20
May 09 12:08:04 pingu kernel:  ? ret_from_fork+0x2d/0x50
May 09 12:08:04 pingu kernel:  ? kthread_complete_and_exit+0x20/0x20
May 09 12:08:04 pingu kernel:  ? ret_from_fork_asm+0x11/0x20
May 09 12:08:04 pingu kernel:  </TASK>
May 09 12:08:04 pingu kernel: Modules linked in: xt_nat macvtap macvlan rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace netfs vhost_net vhost vhost_iotlb tap iptable_nat iptable_filter veth tcp_diag udp_diag inet_diag nf_conntrack_netlink xfrm_user xt_addrtype br_netfi
lter xt_CHECKSUM xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_chain_nat nf_nat bridge stp llc xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables libcrc32c nfnetlink nvme_fabrics overlay bonding tls cfg80211 sunrpc binfmt_misc amdgpu drm
_exec amdxcp drm_buddy gpu_sched intel_rapl_msr intel_rapl_common edac_mce_amd kvm_amd kvm irqbypass crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 snd_hda_codec_hdmi sha1_ssse3 radeon aesni_intel nls_iso8859_1 snd_hda_inte
l snd_intel_dspcfg crypto_simd snd_intel_sdw_acpi drm_suballoc_helper cryptd drm_ttm_helper snd_hda_codec ttm snd_hda_core drm_display_helper snd_pcsp snd_hwdep cec snd_pcm rc_core rapl snd_timer drm_kms_helper wmi_bmof snd video
May 09 12:08:04 pingu kernel:  soundcore ccp mxm_wmi sp5100_tco k10temp joydev mac_hid nct6775 nct6775_core hwmon_vid drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 zfs(PO) spl(O) hid_generic usbhid hid crc32_pclmul ixgbe igb ahci xfrm_algo i2c_algo_bit mdio libahci
 dca xhci_pci xhci_pci_renesas i2c_piix4 wmi
May 09 12:08:04 pingu kernel: CR2: 0000000200000000
May 09 12:08:04 pingu kernel: ---[ end trace 0000000000000000 ]---
May 09 12:08:04 pingu kernel: RIP: 0010:0x200000000
May 09 12:08:04 pingu kernel: Code: Unable to access opcode bytes at 0x1ffffffd6.
May 09 12:08:04 pingu kernel: RSP: 0018:ffffb59e438cfd08 EFLAGS: 00010246
May 09 12:08:04 pingu kernel: RAX: 0000000200000000 RBX: ffff9c0fab2f0000 RCX: 0000000000000000
May 09 12:08:04 pingu kernel: RDX: 0000000000000001 RSI: ffffb59e438cfd58 RDI: ffff9bfb360f7180
May 09 12:08:04 pingu kernel: RBP: ffffb59e438cfdbc R08: 000000000000fdd9 R09: ffff9c38fe7ba4c0
May 09 12:08:04 pingu kernel: R10: 000000000000050a R11: 0000000000000066 R12: 000000000000fdd9
May 09 12:08:04 pingu kernel: R13: ffff9bfaf9386000 R14: ffff9c0fab2f00f8 R15: ffff9bfb360f7180
May 09 12:08:04 pingu kernel: FS:  0000000000000000(0000) GS:ffff9c38fe780000(0000) knlGS:0000000000000000
May 09 12:08:04 pingu kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 09 12:08:04 pingu kernel: CR2: 0000000200000000 CR3: 000000027fb1a000 CR4: 0000000000350ef0
May 09 12:08:04 pingu kernel: note: arc_prune[1366] exited with irqs disabled

An example of iptables on the same system, same kernel:

May 09 15:03:05 pingu kernel: BUG: kernel NULL pointer dereference, address: 0000000000000015
May 09 15:03:05 pingu kernel: #PF: supervisor write access in kernel mode
May 09 15:03:05 pingu kernel: #PF: error_code(0x0002) - not-present page
May 09 15:03:05 pingu kernel: PGD 0 P4D 0 
May 09 15:03:05 pingu kernel: Oops: 0002 [#1] PREEMPT SMP NOPTI
May 09 15:03:05 pingu kernel: CPU: 65 PID: 232279 Comm: iptables Tainted: P           O       6.8.9 #1
May 09 15:03:05 pingu kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C60/TRX40 PRO 10G (MS-7C60), BIOS 1.60 05/13/2020
May 09 15:03:05 pingu kernel: RIP: 0010:iptable_nat_table_init+0xe6/0x170 [iptable_nat]
May 09 15:03:05 pingu kernel: Code: 48 89 ef 48 89 0c 24 e8 98 e9 d7 ff 48 8b 0c 24 85 c0 41 89 c4 75 30 41 83 c7 01 48 83 c1 28 41 83 ff 04 75 d4 48 8b 44 24 08 <4c> 89 30 4c 89 ef e8 3f ec 3c ee 48 83 c4 10 44 89 e0 5b 5d 41 5c
May 09 15:03:05 pingu kernel: RSP: 0018:ffffb71601a93bc8 EFLAGS: 00010246
May 09 15:03:05 pingu kernel: RAX: 0000000000000015 RBX: ffff99c83a710120 RCX: ffff99bba6d73960
May 09 15:03:05 pingu kernel: RDX: 0000000000000000 RSI: 0000000000000064 RDI: ffffffffc1edf1e0
May 09 15:03:05 pingu kernel: RBP: ffff99bd0a29ec00 R08: ffff99f47efd3000 R09: 0000000000000000
May 09 15:03:05 pingu kernel: R10: ffffb71601a93b20 R11: ffff99b696c19088 R12: 0000000000000000
May 09 15:03:05 pingu kernel: R13: ffff99cc9cd6a400 R14: ffff99bba6d738c0 R15: 0000000000000004
May 09 15:03:05 pingu kernel: FS:  00007f9c8074bb48(0000) GS:ffff99f37de40000(0000) knlGS:0000000000000000
May 09 15:03:05 pingu kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 09 15:03:05 pingu kernel: CR2: 0000000000000015 CR3: 00000017f53e2000 CR4: 0000000000350ef0
May 09 15:03:05 pingu kernel: Call Trace:
May 09 15:03:05 pingu kernel:  <TASK>
May 09 15:03:05 pingu kernel:  ? __die_body+0x1b/0x60
May 09 15:03:05 pingu kernel:  ? page_fault_oops+0x15d/0x470
May 09 15:03:05 pingu kernel:  ? do_user_addr_fault+0x65/0x830
May 09 15:03:05 pingu kernel:  ? __kmalloc_node+0x3d3/0x3e0
May 09 15:03:05 pingu kernel:  ? __nf_hook_entries_try_shrink+0x140/0x140
May 09 15:03:05 pingu kernel:  ? exc_page_fault+0x74/0x170
May 09 15:03:05 pingu kernel:  ? asm_exc_page_fault+0x22/0x30
May 09 15:03:05 pingu kernel:  ? iptable_nat_table_init+0xe6/0x170 [iptable_nat]
May 09 15:03:05 pingu kernel:  ? iptable_nat_table_init+0xc8/0x170 [iptable_nat]
May 09 15:03:05 pingu kernel:  xt_find_table_lock+0x128/0x1b0 [x_tables]
May 09 15:03:05 pingu kernel:  xt_request_find_table_lock+0x1b/0x70 [x_tables]
May 09 15:03:05 pingu kernel:  get_info+0x82/0x300 [ip_tables]
May 09 15:03:05 pingu kernel:  ? mntput_no_expire+0x4a/0x240
May 09 15:03:05 pingu kernel:  ? __local_bh_enable_ip+0x37/0x80
May 09 15:03:05 pingu kernel:  ? do_ip_getsockopt+0x85d/0xca0
May 09 15:03:05 pingu kernel:  do_ipt_get_ctl+0x6c/0x330 [ip_tables]
May 09 15:03:05 pingu kernel:  ? obj_cgroup_charge+0xf0/0x110
May 09 15:03:05 pingu kernel:  ? kmem_cache_alloc+0x122/0x2a0
May 09 15:03:05 pingu kernel:  nf_getsockopt+0x44/0x70
May 09 15:03:05 pingu kernel:  ip_getsockopt+0x82/0xc0
May 09 15:03:05 pingu kernel:  do_sock_getsockopt+0x9b/0x220
May 09 15:03:05 pingu kernel:  __sys_getsockopt+0x72/0xc0
May 09 15:03:05 pingu kernel:  __x64_sys_getsockopt+0x21/0x30
May 09 15:03:05 pingu kernel:  do_syscall_64+0x44/0xd0
May 09 15:03:05 pingu kernel:  entry_SYSCALL_64_after_hwframe+0x4b/0x53
May 09 15:03:05 pingu kernel: RIP: 0033:0x7f9c806ed484
May 09 15:03:05 pingu kernel: Code: 31 c9 b8 33 00 00 00 0f 05 48 89 c7 e8 da 29 fe ff 5a c3 49 89 ca 50 48 63 d2 48 63 f6 48 63 ff 45 31 c9 b8 37 00 00 00 0f 05 <48> 63 f8 e8 b9 29 fe ff 5a c3 64 48 8b 04 25 00 00 00 00 48 83 78
May 09 15:03:05 pingu kernel: RSP: 002b:00007ffcccafdb40 EFLAGS: 00000246 ORIG_RAX: 0000000000000037
May 09 15:03:05 pingu kernel: RAX: ffffffffffffffda RBX: 00007ffcccafdf18 RCX: 00007f9c806ed484
May 09 15:03:05 pingu kernel: RDX: 0000000000000040 RSI: 0000000000000000 RDI: 0000000000000004
May 09 15:03:05 pingu kernel: RBP: 0000000000000004 R08: 00007ffcccafdb6c R09: 0000000000000000
May 09 15:03:05 pingu kernel: R10: 00007ffcccafdb74 R11: 0000000000000246 R12: 00007ffcccafdb74
May 09 15:03:05 pingu kernel: R13: 0000000000000002 R14: 00007f9c8074b940 R15: 0000000000000000
May 09 15:03:05 pingu kernel:  </TASK>

Jul 06 '24 15:07 TheUbuntuGuy

I have the same problem on the latest Unraid 7.0.0-beta-1 prerelease. kernel: Linux version 6.8.12-Unraid (root@Develop) (gcc (GCC) 13.2.0, GNU ld version 2.42-slack151) #3 SMP PREEMPT_DYNAMIC Tue Jun 18 07:52:57 PDT 2024

Jul 4 21:21:40 Fractal kernel: BUG: unable to handle page fault for address: 0000000200000002 Jul 4 21:21:40 Fractal kernel: #PF: supervisor instruction fetch in kernel mode Jul 4 21:21:40 Fractal kernel: #PF: error_code(0x0010) - not-present page Jul 4 21:21:40 Fractal kernel: PGD 24af2f067 P4D 24af2f067 PUD 0 Jul 4 21:21:40 Fractal kernel: Oops: 0010 [#1] PREEMPT SMP NOPTI Jul 4 21:21:40 Fractal kernel: CPU: 5 PID: 1324 Comm: arc_prune Tainted: P O 6.8.12-Unraid #3 Jul 4 21:21:40 Fractal kernel: Hardware name: ASUS System Product Name/PRIME H510M-E, BIOS 2402 12/18/2023 Jul 4 21:21:40 Fractal kernel: RIP: 0010:0x200000002 Jul 4 21:21:40 Fractal kernel: Code: Unable to access opcode bytes at 0x1ffffffd8. Jul 4 21:21:40 Fractal kernel: RSP: 0018:ffffc9000098fd30 EFLAGS: 00010246 Jul 4 21:21:40 Fractal kernel: RAX: 0000000200000002 RBX: ffff8884f4070000 RCX: 0000000000000011 Jul 4 21:21:40 Fractal kernel: RDX: ffffffffa0cc54b8 RSI: ffffc9000098fd68 RDI: ffff8881c13ac580 Jul 4 21:21:40 Fractal kernel: RBP: ffffc9000098fdcc R08: 0000000000000000 R09: 00000000001d001c Jul 4 21:21:40 Fractal kernel: R10: 0000000000000000 R11: ffffc9002186fee8 R12: 000000000000bbda Jul 4 21:21:40 Fractal kernel: R13: ffff8881c13ac580 R14: ffff8881c84bfc00 R15: ffff88811176a100 Jul 4 21:21:40 Fractal kernel: FS: 0000000000000000(0000) GS:ffff88883e740000(0000) knlGS:0000000000000000 Jul 4 21:21:40 Fractal kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 4 21:21:40 Fractal kernel: CR2: 0000000200000002 CR3: 0000000251da4006 CR4: 00000000003706f0 Jul 4 21:21:40 Fractal kernel: Call Trace: Jul 4 21:21:40 Fractal kernel: <TASK> Jul 4 21:21:40 Fractal kernel: ? __die_body+0x1a/0x5c Jul 4 21:21:40 Fractal kernel: ? page_fault_oops+0x332/0x37f Jul 4 21:21:40 Fractal kernel: ? put_cpu_partial+0x62/0x8e Jul 4 21:21:40 Fractal kernel: ? spl_kmem_cache_free+0x3a/0x180 [spl] Jul 4 21:21:40 Fractal kernel: ? exc_page_fault+0xf9/0x116 Jul 4 21:21:40 Fractal kernel: ? asm_exc_page_fault+0x22/0x30 Jul 4 21:21:40 Fractal kernel: ? zfs_prune+0xec/0x2ec [zfs] Jul 4 21:21:40 Fractal kernel: ? zpl_prune_sb+0x32/0x50 [zfs] Jul 4 21:21:40 Fractal kernel: ? arc_prune_task+0x1b/0x2e [zfs] Jul 4 21:21:40 Fractal kernel: ? taskq_thread+0x2d4/0x3c1 [spl] Jul 4 21:21:40 Fractal kernel: ? __pfx_default_wake_function+0x10/0x10 Jul 4 21:21:40 Fractal kernel: ? __pfx_taskq_thread+0x10/0x10 [spl] Jul 4 21:21:40 Fractal kernel: ? kthread+0xf4/0xff Jul 4 21:21:40 Fractal kernel: ? __pfx_kthread+0x10/0x10 Jul 4 21:21:40 Fractal kernel: ? ret_from_fork+0x21/0x36 Jul 4 21:21:40 Fractal kernel: ? __pfx_kthread+0x10/0x10 Jul 4 21:21:40 Fractal kernel: ? ret_from_fork_asm+0x1b/0x30 Jul 4 21:21:40 Fractal kernel: </TASK> Jul 4 21:21:40 Fractal kernel: Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip_set nf_tables xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter bridge stp llc nfsd auth_rpcgss oid_registry lockd grace sunrpc bluetooth ecdh_generic ecc md_mod tcp_diag inet_diag ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs macvtap macvlan tap intel_rapl_common x86_pkg_temp_thermal i915 intel_powerclamp coretemp zfs(PO) kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 iosf_mbi sha256_ssse3 drm_buddy sha1_ssse3 ttm aesni_intel crypto_simd i2c_algo_bit cryptd drm_display_helper drm_kms_helper input_leds rapl spl(O) mei_hdcp mei_pxp intel_cstate wmi_bmof drm nvme intel_uncore e1000e hid_apple led_class nvme_core mei_me intel_gtt i2c_i801 agpgart i2c_smbus mei ahci Jul 4 21:21:40 Fractal kernel: i2c_core libahci thermal fan tpm_crb video tpm_tis tpm_tis_core tpm wmi backlight button acpi_tad acpi_pad Jul 4 21:21:40 Fractal kernel: CR2: 0000000200000002 Jul 4 21:21:40 Fractal kernel: ---[ end trace 0000000000000000 ]--- Jul 4 21:21:40 Fractal kernel: pstore: backend (efi_pstore) writing error (-5) Jul 4 21:21:40 Fractal kernel: RIP: 0010:0x200000002 Jul 4 21:21:40 Fractal kernel: Code: Unable to access opcode bytes at 0x1ffffffd8. Jul 4 21:21:40 Fractal kernel: RSP: 0018:ffffc9000098fd30 EFLAGS: 00010246 Jul 4 21:21:40 Fractal kernel: RAX: 0000000200000002 RBX: ffff8884f4070000 RCX: 0000000000000011 Jul 4 21:21:40 Fractal kernel: RDX: ffffffffa0cc54b8 RSI: ffffc9000098fd68 RDI: ffff8881c13ac580 Jul 4 21:21:40 Fractal kernel: RBP: ffffc9000098fdcc R08: 0000000000000000 R09: 00000000001d001c Jul 4 21:21:40 Fractal kernel: R10: 0000000000000000 R11: ffffc9002186fee8 R12: 000000000000bbda Jul 4 21:21:40 Fractal kernel: R13: ffff8881c13ac580 R14: ffff8881c84bfc00 R15: ffff88811176a100 Jul 4 21:21:40 Fractal kernel: FS: 0000000000000000(0000) GS:ffff88883e740000(0000) knlGS:0000000000000000 Jul 4 21:21:40 Fractal kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 4 21:21:40 Fractal kernel: CR2: 0000000200000002 CR3: 0000000251da4006 CR4: 00000000003706f0 Jul 4 21:21:40 Fractal kernel: note: arc_prune[1324] exited with irqs disabled Jul 4 21:21:50 Fractal kernel: veth95a1b70: renamed from eth0

Jul 06 '24 16:07 bpwats

I'm a mod at the Unraid forums and we have seen multiple users with this issue with Docker on zfs since kernel 6.8 (openzfs 2.2.4-1), there was also one report with kernel 6.7 during beta testing, call traces look all very similar, some example in case it helps.

Jul  4 21:21:40 Fractal kernel: BUG: unable to handle page fault for address: 0000000200000002
Jul  4 21:21:40 Fractal kernel: #PF: supervisor instruction fetch in kernel mode
Jul  4 21:21:40 Fractal kernel: #PF: error_code(0x0010) - not-present page
Jul  4 21:21:40 Fractal kernel: PGD 24af2f067 P4D 24af2f067 PUD 0 
Jul  4 21:21:40 Fractal kernel: Oops: 0010 [#1] PREEMPT SMP NOPTI
Jul  4 21:21:40 Fractal kernel: CPU: 5 PID: 1324 Comm: arc_prune Tainted: P           O       6.8.12-Unraid #3
Jul  4 21:21:40 Fractal kernel: Hardware name: ASUS System Product Name/PRIME H510M-E, BIOS 2402 12/18/2023
Jul  4 21:21:40 Fractal kernel: RIP: 0010:0x200000002
Jul  4 21:21:40 Fractal kernel: Code: Unable to access opcode bytes at 0x1ffffffd8.
Jul  4 21:21:40 Fractal kernel: RSP: 0018:ffffc9000098fd30 EFLAGS: 00010246
Jul  4 21:21:40 Fractal kernel: RAX: 0000000200000002 RBX: ffff8884f4070000 RCX: 0000000000000011
Jul  4 21:21:40 Fractal kernel: RDX: ffffffffa0cc54b8 RSI: ffffc9000098fd68 RDI: ffff8881c13ac580
Jul  4 21:21:40 Fractal kernel: RBP: ffffc9000098fdcc R08: 0000000000000000 R09: 00000000001d001c
Jul  4 21:21:40 Fractal kernel: R10: 0000000000000000 R11: ffffc9002186fee8 R12: 000000000000bbda
Jul  4 21:21:40 Fractal kernel: R13: ffff8881c13ac580 R14: ffff8881c84bfc00 R15: ffff88811176a100
Jul  4 21:21:40 Fractal kernel: FS:  0000000000000000(0000) GS:ffff88883e740000(0000) knlGS:0000000000000000
Jul  4 21:21:40 Fractal kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul  4 21:21:40 Fractal kernel: CR2: 0000000200000002 CR3: 0000000251da4006 CR4: 00000000003706f0
Jul  4 21:21:40 Fractal kernel: Call Trace:
Jul  4 21:21:40 Fractal kernel: <TASK>
Jul  4 21:21:40 Fractal kernel: ? __die_body+0x1a/0x5c
Jul  4 21:21:40 Fractal kernel: ? page_fault_oops+0x332/0x37f
Jul  4 21:21:40 Fractal kernel: ? put_cpu_partial+0x62/0x8e
Jul  4 21:21:40 Fractal kernel: ? spl_kmem_cache_free+0x3a/0x180 [spl]
Jul  4 21:21:40 Fractal kernel: ? exc_page_fault+0xf9/0x116
Jul  4 21:21:40 Fractal kernel: ? asm_exc_page_fault+0x22/0x30
Jul  4 21:21:40 Fractal kernel: ? zfs_prune+0xec/0x2ec [zfs]
Jul  4 21:21:40 Fractal kernel: ? zpl_prune_sb+0x32/0x50 [zfs]
Jul  4 21:21:40 Fractal kernel: ? arc_prune_task+0x1b/0x2e [zfs]
Jul  4 21:21:40 Fractal kernel: ? taskq_thread+0x2d4/0x3c1 [spl]
Jul  4 21:21:40 Fractal kernel: ? __pfx_default_wake_function+0x10/0x10
Jul  4 21:21:40 Fractal kernel: ? __pfx_taskq_thread+0x10/0x10 [spl]
Jul  4 21:21:40 Fractal kernel: ? kthread+0xf4/0xff
Jul  4 21:21:40 Fractal kernel: ? __pfx_kthread+0x10/0x10
Jul  4 21:21:40 Fractal kernel: ? ret_from_fork+0x21/0x36
Jul  4 21:21:40 Fractal kernel: ? __pfx_kthread+0x10/0x10
Jul  4 21:21:40 Fractal kernel: ? ret_from_fork_asm+0x1b/0x30
Jul  4 21:21:40 Fractal kernel: </TASK>
Jul  4 21:21:40 Fractal kernel: Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip_set nf_tables xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter bridge stp llc nfsd auth_rpcgss oid_registry lockd grace sunrpc bluetooth ecdh_generic ecc md_mod tcp_diag inet_diag ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs macvtap macvlan tap intel_rapl_common x86_pkg_temp_thermal i915 intel_powerclamp coretemp zfs(PO) kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 iosf_mbi sha256_ssse3 drm_buddy sha1_ssse3 ttm aesni_intel crypto_simd i2c_algo_bit cryptd drm_display_helper drm_kms_helper input_leds rapl spl(O) mei_hdcp mei_pxp intel_cstate wmi_bmof drm nvme intel_uncore e1000e hid_apple led_class nvme_core mei_me intel_gtt i2c_i801 agpgart i2c_smbus mei ahci
Jul  4 21:21:40 Fractal kernel: i2c_core libahci thermal fan tpm_crb video tpm_tis tpm_tis_core tpm wmi backlight button acpi_tad acpi_pad
Jul  4 21:21:40 Fractal kernel: CR2: 0000000200000002
Jul  4 21:21:40 Fractal kernel: ---[ end trace 0000000000000000 ]---
Jul  4 21:21:40 Fractal kernel: pstore: backend (efi_pstore) writing error (-5)
Jul  4 21:21:40 Fractal kernel: RIP: 0010:0x200000002
Jul  4 21:21:40 Fractal kernel: Code: Unable to access opcode bytes at 0x1ffffffd8.
Jul  4 21:21:40 Fractal kernel: RSP: 0018:ffffc9000098fd30 EFLAGS: 00010246
Jul  4 21:21:40 Fractal kernel: RAX: 0000000200000002 RBX: ffff8884f4070000 RCX: 0000000000000011
Jul  4 21:21:40 Fractal kernel: RDX: ffffffffa0cc54b8 RSI: ffffc9000098fd68 RDI: ffff8881c13ac580
Jul  4 21:21:40 Fractal kernel: RBP: ffffc9000098fdcc R08: 0000000000000000 R09: 00000000001d001c
Jul  4 21:21:40 Fractal kernel: R10: 0000000000000000 R11: ffffc9002186fee8 R12: 000000000000bbda
Jul  4 21:21:40 Fractal kernel: R13: ffff8881c13ac580 R14: ffff8881c84bfc00 R15: ffff88811176a100
Jul  4 21:21:40 Fractal kernel: FS:  0000000000000000(0000) GS:ffff88883e740000(0000) knlGS:0000000000000000
Jul  4 21:21:40 Fractal kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul  4 21:21:40 Fractal kernel: CR2: 0000000200000002 CR3: 0000000251da4006 CR4: 00000000003706f0
Jul  4 21:21:40 Fractal kernel: note: arc_prune[1324] exited with irqs disabled

Jul  5 06:58:09 Sirius kernel: BUG: unable to handle page fault for address: 0000000200000000
Jul  5 06:58:09 Sirius kernel: #PF: supervisor instruction fetch in kernel mode
Jul  5 06:58:09 Sirius kernel: #PF: error_code(0x0010) - not-present page
Jul  5 06:58:09 Sirius kernel: PGD 8000000170a48067 P4D 8000000170a48067 PUD 0 
Jul  5 06:58:09 Sirius kernel: Oops: 0010 [#1] PREEMPT SMP PTI
Jul  5 06:58:09 Sirius kernel: CPU: 3 PID: 1079 Comm: arc_prune Tainted: P     U     O       6.8.12-Unraid #3
Jul  5 06:58:09 Sirius kernel: Hardware name: Gigabyte Technology Co., Ltd. C246N-WU2/C246N-WU2-CF, BIOS F2 11/09/2021
Jul  5 06:58:09 Sirius kernel: RIP: 0010:0x200000000
Jul  5 06:58:09 Sirius kernel: Code: Unable to access opcode bytes at 0x1ffffffd6.
Jul  5 06:58:09 Sirius kernel: RSP: 0018:ffffc900005c7d30 EFLAGS: 00010246
Jul  5 06:58:09 Sirius kernel: RAX: 0000000200000000 RBX: ffff8883cb33a000 RCX: 0000000000000011
Jul  5 06:58:09 Sirius kernel: RDX: ffffffffa0fe34b8 RSI: ffffc900005c7d68 RDI: ffff8882ad5d7380
Jul  5 06:58:09 Sirius kernel: RBP: ffffc900005c7dcc R08: 0000000000000000 R09: 0000000000000000
Jul  5 06:58:09 Sirius kernel: R10: 0000000000017c78 R11: 000000000000f40e R12: 000000000000cb06
Jul  5 06:58:09 Sirius kernel: R13: ffff8882ad5d7380 R14: ffff8881363196c0 R15: ffff8881060c8000
Jul  5 06:58:09 Sirius kernel: FS:  0000000000000000(0000) GS:ffff88884e580000(0000) knlGS:0000000000000000
Jul  5 06:58:09 Sirius kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul  5 06:58:09 Sirius kernel: CR2: 0000000200000000 CR3: 0000000170508004 CR4: 00000000003706f0
Jul  5 06:58:09 Sirius kernel: Call Trace:
Jul  5 06:58:09 Sirius kernel: <TASK>
Jul  5 06:58:09 Sirius kernel: ? __die_body+0x1a/0x5c
Jul  5 06:58:09 Sirius kernel: ? page_fault_oops+0x332/0x37f
Jul  5 06:58:09 Sirius kernel: ? call_rcu+0x530/0x5e6
Jul  5 06:58:09 Sirius kernel: ? exc_page_fault+0xf9/0x116
Jul  5 06:58:09 Sirius kernel: ? asm_exc_page_fault+0x22/0x30
Jul  5 06:58:09 Sirius kernel: ? zfs_prune+0xec/0x2ec [zfs]
Jul  5 06:58:09 Sirius kernel: ? __schedule+0x69c/0x6e8
Jul  5 06:58:09 Sirius kernel: ? zpl_prune_sb+0x32/0x50 [zfs]
Jul  5 06:58:09 Sirius kernel: ? arc_prune_task+0x1b/0x2e [zfs]
Jul  5 06:58:09 Sirius kernel: ? taskq_thread+0x2d4/0x3c1 [spl]
Jul  5 06:58:09 Sirius kernel: ? __pfx_default_wake_function+0x10/0x10
Jul  5 06:58:09 Sirius kernel: ? __pfx_taskq_thread+0x10/0x10 [spl]
Jul  5 06:58:09 Sirius kernel: ? kthread+0xf4/0xff
Jul  5 06:58:09 Sirius kernel: ? __pfx_kthread+0x10/0x10
Jul  5 06:58:09 Sirius kernel: ? ret_from_fork+0x21/0x36
Jul  5 06:58:09 Sirius kernel: ? __pfx_kthread+0x10/0x10
Jul  5 06:58:09 Sirius kernel: ? ret_from_fork_asm+0x1b/0x30
Jul  5 06:58:09 Sirius kernel: </TASK>
Jul  5 06:58:09 Sirius kernel: Modules linked in: bluetooth ecdh_generic ecc xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter md_mod tcp_diag inet_diag kvmgt mdev i915 drm_buddy ttm drm_display_helper drm_kms_helper drm intel_gtt agpgart ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls e1000e igb i2c_algo_bit intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp zfs(PO) kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd mei_pxp mei_hdcp rapl spl(O) gigabyte_wmi wmi_bmof i2c_i801 mei_me intel_cstate intel_uncore i2c_smbus nvme mei joydev i2c_core ahci input_leds led_class nvme_core libahci intel_pch_thermal fan thermal video tpm_crb wmi backlight tpm_tis tpm_tis_core tpm button acpi_pad [last unloaded: e1000e]
Jul  5 06:58:09 Sirius kernel: CR2: 0000000200000000
Jul  5 06:58:09 Sirius kernel: ---[ end trace 0000000000000000 ]---
Jul  5 06:58:09 Sirius kernel: RIP: 0010:0x200000000
Jul  5 06:58:09 Sirius kernel: Code: Unable to access opcode bytes at 0x1ffffffd6.
Jul  5 06:58:09 Sirius kernel: RSP: 0018:ffffc900005c7d30 EFLAGS: 00010246
Jul  5 06:58:09 Sirius kernel: RAX: 0000000200000000 RBX: ffff8883cb33a000 RCX: 0000000000000011
Jul  5 06:58:09 Sirius kernel: RDX: ffffffffa0fe34b8 RSI: ffffc900005c7d68 RDI: ffff8882ad5d7380
Jul  5 06:58:09 Sirius kernel: RBP: ffffc900005c7dcc R08: 0000000000000000 R09: 0000000000000000
Jul  5 06:58:09 Sirius kernel: R10: 0000000000017c78 R11: 000000000000f40e R12: 000000000000cb06
Jul  5 06:58:09 Sirius kernel: R13: ffff8882ad5d7380 R14: ffff8881363196c0 R15: ffff8881060c8000
Jul  5 06:58:09 Sirius kernel: FS:  0000000000000000(0000) GS:ffff88884e580000(0000) knlGS:0000000000000000
Jul  5 06:58:09 Sirius kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul  5 06:58:09 Sirius kernel: CR2: 0000000200000000 CR3: 0000000170508004 CR4: 00000000003706f0
Jul  5 06:58:09 Sirius kernel: note: arc_prune[1079] exited with irqs disabled

Jul  1 09:09:32 SofaKing kernel: BUG: unable to handle page fault for address: 0000000200000000
Jul  1 09:09:32 SofaKing kernel: #PF: supervisor instruction fetch in kernel mode
Jul  1 09:09:32 SofaKing kernel: #PF: error_code(0x0010) - not-present page
Jul  1 09:09:32 SofaKing kernel: PGD 0 P4D 0 
Jul  1 09:09:32 SofaKing kernel: Oops: 0010 [#1] PREEMPT SMP NOPTI
Jul  1 09:09:32 SofaKing kernel: CPU: 8 PID: 1547 Comm: arc_prune Tainted: P           O       6.8.12-Unraid #3
Jul  1 09:09:32 SofaKing kernel: Hardware name: System manufacturer System Product Name/TUF GAMING X570-PLUS (WI-FI), BIOS 4602 02/23/2023
Jul  1 09:09:32 SofaKing kernel: RIP: 0010:0x200000000
Jul  1 09:09:32 SofaKing kernel: Code: Unable to access opcode bytes at 0x1ffffffd6.
Jul  1 09:09:32 SofaKing kernel: RSP: 0018:ffffc90001267d30 EFLAGS: 00010246
Jul  1 09:09:32 SofaKing kernel: RAX: 0000000200000000 RBX: ffff8884254b0000 RCX: 0000000000000011
Jul  1 09:09:32 SofaKing kernel: RDX: ffffffffa43294b8 RSI: ffffc90001267d68 RDI: ffff88866784b780
Jul  1 09:09:32 SofaKing kernel: RBP: ffffc90001267dcc R08: 0000000000000000 R09: 00000000001d001c
Jul  1 09:09:32 SofaKing kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000c1d1
Jul  1 09:09:32 SofaKing kernel: R13: ffff88866784b780 R14: ffff88810160a0c0 R15: ffff88811025e180
Jul  1 09:09:32 SofaKing kernel: FS:  0000000000000000(0000) GS:ffff888feea00000(0000) knlGS:0000000000000000
Jul  1 09:09:32 SofaKing kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul  1 09:09:32 SofaKing kernel: CR2: 0000000200000000 CR3: 0000000210406000 CR4: 0000000000750ef0
Jul  1 09:09:32 SofaKing kernel: PKRU: 55555554
Jul  1 09:09:32 SofaKing kernel: Call Trace:
Jul  1 09:09:32 SofaKing kernel: <TASK>
Jul  1 09:09:32 SofaKing kernel: ? __die_body+0x1a/0x5c
Jul  1 09:09:32 SofaKing kernel: ? page_fault_oops+0x332/0x37f
Jul  1 09:09:32 SofaKing kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Jul  1 09:09:32 SofaKing kernel: ? spl_kmem_cache_free+0x3a/0x180 [spl]
Jul  1 09:09:32 SofaKing kernel: ? exc_page_fault+0xf9/0x116
Jul  1 09:09:32 SofaKing kernel: ? asm_exc_page_fault+0x22/0x30
Jul  1 09:09:32 SofaKing kernel: ? zfs_prune+0xef/0x2ec [zfs]
Jul  1 09:09:32 SofaKing kernel: ? zpl_prune_sb+0x32/0x50 [zfs]
Jul  1 09:09:32 SofaKing kernel: ? arc_prune_task+0x1e/0x2e [zfs]
Jul  1 09:09:32 SofaKing kernel: ? taskq_thread+0x2d7/0x3c1 [spl]
Jul  1 09:09:32 SofaKing kernel: ? __pfx_default_wake_function+0x10/0x10
Jul  1 09:09:32 SofaKing kernel: ? __pfx_taskq_thread+0x10/0x10 [spl]
Jul  1 09:09:32 SofaKing kernel: ? kthread+0xf7/0xff
Jul  1 09:09:32 SofaKing kernel: ? __pfx_kthread+0x10/0x10
Jul  1 09:09:32 SofaKing kernel: ? ret_from_fork+0x24/0x36
Jul  1 09:09:32 SofaKing kernel: ? __pfx_kthread+0x10/0x10
Jul  1 09:09:32 SofaKing kernel: ? ret_from_fork_asm+0x1b/0x30
Jul  1 09:09:32 SofaKing kernel: </TASK>
Jul  1 09:09:32 SofaKing kernel: Modules linked in: xt_nat veth nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter xt_CHECKSUM xt_conntrack ipt_REJECT nf_reject_ipv4 ip6table_mangle iptable_mangle vhost_net vhost vhost_iotlb nvidia_uvm(PO) nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod xt_tcpudp xt_mark tun nf_tables nfnetlink ip6table_nat tcp_diag inet_diag nct6775 nct6775_core hwmon_vid iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs 8021q garp mrp macvtap macvlan tap bridge stp llc atlantic r8169 realtek edac_mce_amd edac_core intel_rapl_common iosf_mbi nvidia_drm(PO) nvidia_modeset(PO) kvm_amd zfs(PO) nvidia(PO) video kvm drm_kms_helper spl(O) crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel btusb sha512_ssse3 drm sha256_ssse3 sha1_ssse3 btrtl aesni_intel btbcm
Jul  1 09:09:32 SofaKing kernel: crypto_simd btintel cp210x cryptd wmi_bmof rapl joydev bluetooth input_leds acpi_cpufreq i2c_piix4 backlight k10temp usbserial nvme ccp ahci i2c_core ecdh_generic libahci ecc nvme_core led_class wmi tpm_crb tpm_tis tpm_tis_core tpm button [last unloaded: atlantic]
Jul  1 09:09:32 SofaKing kernel: CR2: 0000000200000000
Jul  1 09:09:32 SofaKing kernel: ---[ end trace 0000000000000000 ]---
Jul  1 09:09:32 SofaKing kernel: RIP: 0010:0x200000000
Jul  1 09:09:32 SofaKing kernel: Code: Unable to access opcode bytes at 0x1ffffffd6.
Jul  1 09:09:32 SofaKing kernel: RSP: 0018:ffffc90001267d30 EFLAGS: 00010246
Jul  1 09:09:32 SofaKing kernel: RAX: 0000000200000000 RBX: ffff8884254b0000 RCX: 0000000000000011
Jul  1 09:09:32 SofaKing kernel: RDX: ffffffffa43294b8 RSI: ffffc90001267d68 RDI: ffff88866784b780
Jul  1 09:09:32 SofaKing kernel: RBP: ffffc90001267dcc R08: 0000000000000000 R09: 00000000001d001c
Jul  1 09:09:32 SofaKing kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000c1d1
Jul  1 09:09:32 SofaKing kernel: R13: ffff88866784b780 R14: ffff88810160a0c0 R15: ffff88811025e180
Jul  1 09:09:32 SofaKing kernel: FS:  0000000000000000(0000) GS:ffff888feea00000(0000) knlGS:0000000000000000
Jul  1 09:09:32 SofaKing kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul  1 09:09:32 SofaKing kernel: CR2: 0000000200000000 CR3: 0000000210406000 CR4: 0000000000750ef0
Jul  1 09:09:32 SofaKing kernel: PKRU: 55555554
Jul  1 09:09:32 SofaKing kernel: note: arc_prune[1547] exited with irqs disabled

Jun 30 03:00:45 unRAID kernel: BUG: unable to handle page fault for address: 0000000200000002
Jun 30 03:00:45 unRAID kernel: #PF: supervisor instruction fetch in kernel mode
Jun 30 03:00:45 unRAID kernel: #PF: error_code(0x0010) - not-present page
Jun 30 03:00:45 unRAID kernel: PGD c1420c067 P4D c1420c067 PUD 0 
Jun 30 03:00:45 unRAID kernel: Oops: 0010 [#1] PREEMPT SMP NOPTI
Jun 30 03:00:45 unRAID kernel: CPU: 3 PID: 9677 Comm: arc_prune Tainted: P           O       6.8.12-Unraid #3
Jun 30 03:00:45 unRAID kernel: Hardware name: ASRock Z690 PG Riptide/Z690 PG Riptide, BIOS 18.04 06/07/2024
Jun 30 03:00:45 unRAID kernel: RIP: 0010:0x200000002
Jun 30 03:00:45 unRAID kernel: Code: Unable to access opcode bytes at 0x1ffffffd8.
Jun 30 03:00:45 unRAID kernel: RSP: 0018:ffffc9000086bd30 EFLAGS: 00010246
Jun 30 03:00:45 unRAID kernel: RAX: 0000000200000002 RBX: ffff8885cf7f4000 RCX: 0000000000000011
Jun 30 03:00:45 unRAID kernel: RDX: ffffffffa0fd84b8 RSI: ffffc9000086bd68 RDI: ffff888294f3f780
Jun 30 03:00:45 unRAID kernel: RBP: ffffc9000086bdcc R08: 0000000000000000 R09: ffff88810b63f548
Jun 30 03:00:45 unRAID kernel: R10: 000000000000065a R11: 0000000000000699 R12: 000000000002040e
Jun 30 03:00:45 unRAID kernel: R13: ffff888294f3f780 R14: ffff88813cdfbe00 R15: ffff88813beba080
Jun 30 03:00:45 unRAID kernel: FS:  0000000000000000(0000) GS:ffff88904f2c0000(0000) knlGS:0000000000000000
Jun 30 03:00:45 unRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 30 03:00:45 unRAID kernel: CR2: 0000000200000002 CR3: 00000002c443a000 CR4: 0000000000752ef0
Jun 30 03:00:45 unRAID kernel: PKRU: 55555554
Jun 30 03:00:45 unRAID kernel: Call Trace:
Jun 30 03:00:45 unRAID kernel: <TASK>
Jun 30 03:00:45 unRAID kernel: ? __die_body+0x1a/0x5c
Jun 30 03:00:45 unRAID kernel: ? page_fault_oops+0x332/0x37f
Jun 30 03:00:45 unRAID kernel: ? call_rcu+0x530/0x5e6
Jun 30 03:00:45 unRAID kernel: ? exc_page_fault+0xf9/0x116
Jun 30 03:00:45 unRAID kernel: ? asm_exc_page_fault+0x22/0x30
Jun 30 03:00:45 unRAID kernel: ? zfs_prune+0xec/0x2ec [zfs]
Jun 30 03:00:45 unRAID kernel: ? autoremove_wake_function+0xe/0x33
Jun 30 03:00:45 unRAID kernel: ? zpl_prune_sb+0x32/0x50 [zfs]
Jun 30 03:00:45 unRAID kernel: ? arc_prune_task+0x1b/0x2e [zfs]
Jun 30 03:00:45 unRAID kernel: ? taskq_thread+0x2d4/0x3c1 [spl]
Jun 30 03:00:45 unRAID kernel: ? __pfx_default_wake_function+0x10/0x10
Jun 30 03:00:45 unRAID kernel: ? __pfx_taskq_thread+0x10/0x10 [spl]
Jun 30 03:00:45 unRAID kernel: ? kthread+0xf4/0xff
Jun 30 03:00:45 unRAID kernel: ? __pfx_kthread+0x10/0x10
Jun 30 03:00:45 unRAID kernel: ? ret_from_fork+0x21/0x36
Jun 30 03:00:45 unRAID kernel: ? __pfx_kthread+0x10/0x10
Jun 30 03:00:45 unRAID kernel: ? ret_from_fork_asm+0x1b/0x30
Jun 30 03:00:45 unRAID kernel: </TASK>
Jun 30 03:00:45 unRAID kernel: Modules linked in: veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_nat xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter bridge dm_crypt dm_mod nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) spl(O) tcp_diag inet_diag nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs macvtap macvlan tap 8021q garp mrp stp llc mlx4_en xe drm_gpuvm drm_exec gpu_sched drm_ttm_helper drm_suballoc_helper x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel i915 kvm drm_buddy crct10dif_pclmul crc32_pclmul ttm crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 i2c_algo_bit sha1_ssse3 drm_display_helper aesni_intel crypto_simd cryptd drm_kms_helper mei_hdcp mei_pxp rapl drm processor_thermal_device_pci i2c_i801 mei_me processor_thermal_device intel_cstate
Jun 30 03:00:45 unRAID kernel: processor_thermal_wt_hint wmi_bmof mlx4_core intel_gtt intel_uncore processor_thermal_rfim nvme processor_thermal_rapl intel_rapl_common processor_thermal_wt_req i2c_smbus ahci agpgart mei input_leds processor_thermal_power_floor nvme_core libahci joydev processor_thermal_mbox led_class i2c_core int340x_thermal_zone iosf_mbi tpm_crb tpm_tis video tpm_tis_core wmi tpm int3400_thermal backlight acpi_thermal_rel acpi_pad acpi_tad button
Jun 30 03:00:45 unRAID kernel: CR2: 0000000200000002
Jun 30 03:00:45 unRAID kernel: ---[ end trace 0000000000000000 ]---
Jun 30 03:00:45 unRAID kernel: RIP: 0010:0x200000002
Jun 30 03:00:45 unRAID kernel: Code: Unable to access opcode bytes at 0x1ffffffd8.
Jun 30 03:00:45 unRAID kernel: RSP: 0018:ffffc9000086bd30 EFLAGS: 00010246
Jun 30 03:00:45 unRAID kernel: RAX: 0000000200000002 RBX: ffff8885cf7f4000 RCX: 0000000000000011
Jun 30 03:00:45 unRAID kernel: RDX: ffffffffa0fd84b8 RSI: ffffc9000086bd68 RDI: ffff888294f3f780
Jun 30 03:00:45 unRAID kernel: RBP: ffffc9000086bdcc R08: 0000000000000000 R09: ffff88810b63f548
Jun 30 03:00:45 unRAID kernel: R10: 000000000000065a R11: 0000000000000699 R12: 000000000002040e
Jun 30 03:00:45 unRAID kernel: R13: ffff888294f3f780 R14: ffff88813cdfbe00 R15: ffff88813beba080
Jun 30 03:00:45 unRAID kernel: FS:  0000000000000000(0000) GS:ffff88904f2c0000(0000) knlGS:0000000000000000
Jun 30 03:00:45 unRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 30 03:00:45 unRAID kernel: CR2: 0000000200000002 CR3: 00000002c443a000 CR4: 0000000000752ef0
Jun 30 03:00:45 unRAID kernel: PKRU: 55555554
Jun 30 03:00:45 unRAID kernel: note: arc_prune[9677] exited with irqs disabled

Jul 07 '24 10:07 1JorgeB

We've been hitting what looks to be this issue ever since we launched our new infrastructure all running on the latest Ubuntu Kernel and ZFS. Once we had migrated some hundreds of container workloads we started experiencing crashes. We've been very unfortunate in our crash dump collection, but we've ascertained that the crashes we are seeing is very similar to this, please see the LXCFS issue linked above.

Symptoms as we see them:

Triggered by read-heavy workloads. The more read you have the more likely this is to happen.
Upping the ARC cache to the maximum we are able to, has greatly mitigated these crashes. Seems like reading from ARC in RAM prevents this from happening to some extent. (We also spread out read-heavy workloads as much as we could to different hosts, which also seems to have helped)
The issue seems to affect LXCFS, likely due to memory getting corrupted

ZFS version where we've seen this is zfs-2.2.2-0ubuntu9 to zfs-2.2.4-1 / zfs-kmod-2.2.4-1 (Zabbly) Kernel versions we've seen this is: 6.8.0-31-generic #31-Ubuntu to 6.9.10-zabbly+ #ubuntu24.04

These workloads were all stable for years on older kernels.

This is a real issue and I would not be surprised to learn that a lot of zfs users out there are being affected by this now. It took us a long time to track down the source of our crashes, and I expect others may be in the same situation. I believe this issue warrants immediate attention, especially since upgrading to the latest mainline-ish kernel and zfs does not seem to resolve this.

Jul 23 '24 12:07 webdock-io

Taking into account that this crash happens in a shrinkers-related code I can make a wild guess that this issue should be provokable by something like echo 3 > /proc/sys/vm/drop_caches (don't try on production systems!).

Jul 23 '24 12:07 mihalicyn

Taking into account that this crash happens in a shrinkers-related code I can make a wild guess that this issue should be provokable by something like echo 3 > /proc/sys/vm/drop_caches (don't try on production systems!).

I tried that and it did not crash my system or trigger the issue. AFAIK the ZFS ARC is separate.

My guess would be it's hitting a race condition of sorts on heavy IO where it has to be evicting a lot out of ARC.

Jul 23 '24 22:07 maxpoulin64

I have a similar issue with kernel 6.8.12, docker 27.0.3, and zfs 2.2.4 (zfs-2.2.4-r0-gentoo) on Gentoo. My trace does not include arc_prune but zfs_prune.

[Thu Jul 25 10:11:50 2024] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[Thu Jul 25 10:11:50 2024] BUG: unable to handle page fault for address: ffff888209535180
[Thu Jul 25 10:11:50 2024] #PF: supervisor instruction fetch in kernel mode
[Thu Jul 25 10:11:50 2024] #PF: error_code(0x0011) - permissions violation
[Thu Jul 25 10:11:50 2024] PGD 3001067 P4D 3001067 PUD 81f5f2067 PMD 40b61e063 PTE 8000000209535063
[Thu Jul 25 10:11:50 2024] Oops: 0011 [#1] SMP PTI
[Thu Jul 25 10:11:50 2024] CPU: 2 PID: 3297 Comm: arc_prune Tainted: P           O       6.8.12-gentoo #1
[Thu Jul 25 10:11:50 2024] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z77X-D3H, BIOS F18i 01/06/2014
[Thu Jul 25 10:11:50 2024] RIP: 0010:0xffff888209535180
[Thu Jul 25 10:11:50 2024] Code: 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 85 e8 4b 8d 82 88
[Thu Jul 25 10:11:50 2024] RSP: 0018:ffffc900004ebd20 EFLAGS: 00010246
[Thu Jul 25 10:11:50 2024] RAX: ffff8881031a5c40 RBX: ffff8883be0ce000 RCX: 00000000031a5c31
[Thu Jul 25 10:11:50 2024] RDX: 0000000000000000 RSI: ffffc900004ebd68 RDI: ffff88828d4be880
[Thu Jul 25 10:11:50 2024] RBP: ffff88828d4be880 R08: 0000000000000000 R09: 0000000000000000
[Thu Jul 25 10:11:50 2024] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000206
[Thu Jul 25 10:11:50 2024] R13: ffffc900004ebdcc R14: ffff8883be0ce0f8 R15: ffff888108505200
[Thu Jul 25 10:11:50 2024] FS:  0000000000000000(0000) GS:ffff8887ff300000(0000) knlGS:0000000000000000
[Thu Jul 25 10:11:50 2024] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Thu Jul 25 10:11:50 2024] CR2: ffff888209535180 CR3: 000000000262e001 CR4: 00000000001706f0
[Thu Jul 25 10:11:50 2024] Call Trace:
[Thu Jul 25 10:11:50 2024]  <TASK>
[Thu Jul 25 10:11:50 2024]  ? __die+0x1a/0x60
[Thu Jul 25 10:11:50 2024]  ? page_fault_oops+0x158/0x430
[Thu Jul 25 10:11:50 2024]  ? search_extable+0x22/0x30
[Thu Jul 25 10:11:50 2024]  ? search_module_extables+0x9/0x30
[Thu Jul 25 10:11:50 2024]  ? fixup_exception+0x1d/0x240
[Thu Jul 25 10:11:50 2024]  ? exc_page_fault+0x28a/0x580
[Thu Jul 25 10:11:50 2024]  ? asm_exc_page_fault+0x22/0x30
[Thu Jul 25 10:11:50 2024]  ? zfs_prune+0x9b/0x3f0 [zfs]
[Thu Jul 25 10:11:50 2024]  ? __switch_to_asm+0x3a/0x60
[Thu Jul 25 10:11:50 2024]  ? __switch_to_asm+0x34/0x60
[Thu Jul 25 10:11:50 2024]  ? zpl_prune_sb+0x2f/0x1780 [zfs]
[Thu Jul 25 10:11:50 2024]  ? arc_getbuf_func+0x26/0x340 [zfs]
[Thu Jul 25 10:11:50 2024]  ? taskq_dispatch+0x48f/0x680 [spl]
[Thu Jul 25 10:11:50 2024]  ? wake_up_state+0x10/0x10
[Thu Jul 25 10:11:50 2024]  ? taskq_dispatch+0x240/0x680 [spl]
[Thu Jul 25 10:11:50 2024]  ? kthread+0xc4/0xf0
[Thu Jul 25 10:11:50 2024]  ? kthread_complete_and_exit+0x20/0x20
[Thu Jul 25 10:11:50 2024]  ? ret_from_fork+0x28/0x40
[Thu Jul 25 10:11:50 2024]  ? kthread_complete_and_exit+0x20/0x20
[Thu Jul 25 10:11:50 2024]  ? ret_from_fork_asm+0x11/0x20
[Thu Jul 25 10:11:50 2024]  </TASK>
[Thu Jul 25 10:11:50 2024] Modules linked in: wireguard libchacha20poly1305 chacha_x86_64 poly1305_x86_64 br_netfilter em28xx_rc si2157 si2168 bridge stp llc xt_MASQUERADE xt_addrtype zfs(PO) spl(O) xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 em28xx_alsa ip6table_filter ip6_tables drxk em28xx_dvb snd_hda_codec_hdmi snd_hda_codec_via snd_hda_codec_generic led_class snd_hda_intel adm1021 snd_intel_dspcfg snd_hda_codec em28xx x86_pkg_temp_thermal snd_hda_core i915 cdc_acm tveeprom snd_pcm atl1c mpt3sas snd_timer i2c_algo_bit raid_class drm_buddy scsi_transport_sas drm_display_helper fan ttm video evdev wmi
[Thu Jul 25 10:11:50 2024] CR2: ffff888209535180
[Thu Jul 25 10:11:50 2024] ---[ end trace 0000000000000000 ]---
[Thu Jul 25 10:11:50 2024] RIP: 0010:0xffff888209535180
[Thu Jul 25 10:11:50 2024] Code: 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 85 e8 4b 8d 82 88
[Thu Jul 25 10:11:50 2024] RSP: 0018:ffffc900004ebd20 EFLAGS: 00010246
[Thu Jul 25 10:11:50 2024] RAX: ffff8881031a5c40 RBX: ffff8883be0ce000 RCX: 00000000031a5c31
[Thu Jul 25 10:11:50 2024] RDX: 0000000000000000 RSI: ffffc900004ebd68 RDI: ffff88828d4be880
[Thu Jul 25 10:11:50 2024] RBP: ffff88828d4be880 R08: 0000000000000000 R09: 0000000000000000
[Thu Jul 25 10:11:50 2024] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000206
[Thu Jul 25 10:11:50 2024] R13: ffffc900004ebdcc R14: ffff8883be0ce0f8 R15: ffff888108505200
[Thu Jul 25 10:11:50 2024] FS:  0000000000000000(0000) GS:ffff8887ff300000(0000) knlGS:0000000000000000
[Thu Jul 25 10:11:50 2024] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Thu Jul 25 10:11:50 2024] CR2: ffff888209535180 CR3: 000000000262e001 CR4: 00000000001706f0
[Thu Jul 25 10:11:50 2024] note: arc_prune[3297] exited with irqs disabled

Jul 25 '24 09:07 danieldietsch

@1JorgeB any chance switching storage drivers might be a reliable workaround until this gets resolved, versus using a btrfs image?

Jul 27 '24 21:07 andrebrait

same issue here, on 6.8.9. crashes after an AI training workload for a few hours.

will revert to 6.6.

EDIT: 6.6 is stable (am not using docker, just standard filesystem reading and writing).

Aug 01 '24 03:08 Xynonners

FYI OpenZFS has supported docker's overlay2 storage driver since 2.2. See https://github.com/moby/moby/issues/46337#issuecomment-1899224961

I gave up on the docker zfs storage driver some time ago, as it was pretty buggy.

If you are starting docker with systemd you can modify the startup line to be:

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock -s overlay2

Using overlay2 I don't have any issues loading docker run -it --rm -p 8080:8080 --gpus all --name localai quay.io/go-skynet/local-ai:latest-aio-gpu-hipblas using a kernel newer than 6.9 using zfs built from https://github.com/openzfs/zfs/pull/16359 . In fairness I'm also adding some patches from PRs on top of that so that I can be on kernel 6.11.0-rc1...

Aug 01 '24 20:08 satmandu

But maybe also https://github.com/openzfs/zfs/pull/16401 and https://github.com/openzfs/zfs/pull/16404 might help track down the issue?

Aug 01 '24 20:08 satmandu

Similar situation here, this time with 6.10.4 and zfs 2.2.5-1 on Debian.

This occurred during a docker compose pull while some image layers were being extracted in tandem. I am also using the zfs driver.

I was forced to power cycle after this, which incidentally upgraded my kernel to 6.10.6, and the same pull succeeded fine afterward. But the ARC conditions would also be entirely different after a fresh boot so I doubt the kernel upgrade mattered. Just some info.

FWIW, my kernel is in lockdown mode due to secure boot.

Aug 29 04:25:08 server kernel: PGD 0 P4D 0
Aug 29 04:25:08 server kernel: Oops: Oops: 0010 [#1] PREEMPT SMP NOPTI
Aug 29 04:25:08 server kernel: CPU: 8 PID: 1168 Comm: arc_prune Tainted: P           O       6.10.4-amd64 #1  Debian 6.10.4-1
Aug 29 04:25:08 server kernel: Hardware name: Supermicro Super Server/H12SSL-CT, BIOS 2.8 02/27/2024
Aug 29 04:25:08 server kernel: RIP: 0010:0x200000000
Aug 29 04:25:08 server kernel: Code: Unable to access opcode bytes at 0x1ffffffd6.
Aug 29 04:25:08 server kernel: RSP: 0018:ffffbdfb4be4fce8 EFLAGS: 00010246
Aug 29 04:25:08 server kernel: RAX: 0000000200000000 RBX: ffffbdfb4be4fd9c RCX: 0000000000000000
Aug 29 04:25:08 server kernel: RDX: 0000000000000000 RSI: ffffbdfb4be4fd38 RDI: ffff961b87569400
Aug 29 04:25:08 server kernel: RBP: 0000000000023f7e R08: ffff96279b820000 R09: ffff9618fdb7da28
Aug 29 04:25:08 server kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffff961b87569400
Aug 29 04:25:08 server kernel: R13: ffff96190575ff70 R14: ffff9618fdb7da90 R15: ffff96279b8200f8
Aug 29 04:25:08 server kernel: FS:  0000000000000000(0000) GS:ffff96378dc00000(0000) knlGS:0000000000000000
Aug 29 04:25:08 server kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 29 04:25:08 server kernel: CR2: 0000000200000000 CR3: 0000000f6e890000 CR4: 0000000000350ef0
Aug 29 04:25:08 server kernel: Call Trace:
Aug 29 04:25:08 server kernel:  <TASK>
Aug 29 04:25:08 server kernel:  ? __die+0x23/0x70
Aug 29 04:25:08 server kernel:  ? page_fault_oops+0x173/0x5a0
Aug 29 04:25:08 server kernel:  ? spl_kmem_cache_free+0x130/0x1e0 [spl]
Aug 29 04:25:08 server kernel:  ? exc_page_fault+0x7e/0x180
Aug 29 04:25:08 server kernel:  ? asm_exc_page_fault+0x26/0x30
Aug 29 04:25:08 server kernel:  ? zfs_prune+0xba/0x4e0 [zfs]
Aug 29 04:25:08 server kernel:  ? finish_task_switch.isra.0+0x97/0x2c0
Aug 29 04:25:08 server kernel:  ? srso_return_thunk+0x5/0x5f
Aug 29 04:25:08 server kernel:  ? zpl_prune_sb+0x38/0x60 [zfs]
Aug 29 04:25:08 server kernel:  ? arc_prune_task+0x22/0x40 [zfs]
Aug 29 04:25:08 server kernel:  ? taskq_thread+0x2ba/0x500 [spl]
Aug 29 04:25:08 server kernel:  ? __pfx_default_wake_function+0x10/0x10
Aug 29 04:25:08 server kernel:  ? __pfx_taskq_thread+0x10/0x10 [spl]
Aug 29 04:25:08 server kernel:  ? kthread+0xd2/0x100
Aug 29 04:25:08 server kernel:  ? __pfx_kthread+0x10/0x10
Aug 29 04:25:08 server kernel:  ? ret_from_fork+0x34/0x50
Aug 29 04:25:08 server kernel:  ? __pfx_kthread+0x10/0x10
Aug 29 04:25:08 server kernel:  ? ret_from_fork_asm+0x1a/0x30
Aug 29 04:25:08 server kernel:  </TASK>
Aug 29 04:25:08 server kernel: Modules linked in: cpuid udp_diag tcp_diag inet_diag wireguard libchacha20poly1305 chacha_x86_64 poly1305_x86_64 curve25519_x86_64 libcurve25519_generic libchacha ip>
Aug 29 04:25:08 server kernel:  watchdog k10temp ipmi_msghandler joydev evdev sg nvme_fabrics drm efi_pstore configfs nfnetlink ip_tables x_tables autofs4 zfs(PO) spl(O) efivarfs raid10 raid0 raid>
Aug 29 04:25:08 server kernel: CR2: 0000000200000000
Aug 29 04:25:08 server kernel: ---[ end trace 0000000000000000 ]---
Aug 29 04:25:08 server kernel: RIP: 0010:0x200000000
Aug 29 04:25:08 server kernel: Code: Unable to access opcode bytes at 0x1ffffffd6.
Aug 29 04:25:08 server kernel: RSP: 0018:ffffbdfb4be4fce8 EFLAGS: 00010246
Aug 29 04:25:08 server kernel: RAX: 0000000200000000 RBX: ffffbdfb4be4fd9c RCX: 0000000000000000
Aug 29 04:25:08 server kernel: RDX: 0000000000000000 RSI: ffffbdfb4be4fd38 RDI: ffff961b87569400
Aug 29 04:25:08 server kernel: RBP: 0000000000023f7e R08: ffff96279b820000 R09: ffff9618fdb7da28
Aug 29 04:25:08 server kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffff961b87569400
Aug 29 04:25:08 server kernel: R13: ffff96190575ff70 R14: ffff9618fdb7da90 R15: ffff96279b8200f8
Aug 29 04:25:08 server kernel: FS:  0000000000000000(0000) GS:ffff96378dc00000(0000) knlGS:0000000000000000
Aug 29 04:25:08 server kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 29 04:25:08 server kernel: CR2: 0000000200000000 CR3: 0000000f6e890000 CR4: 0000000000350ef0
Aug 29 04:25:08 server kernel: note: arc_prune[1168] exited with irqs disabled

Aug 29 '24 09:08 putnam

Still reproducing in ZFS 2.2.6 and kernel 6.10.8-zen1-1-zen.

Sep 09 '24 02:09 maxpoulin64

I was playing with this one today and trying to reproduce it:

# uname -a
Linux hoodin 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

# modinfo zfs
filename:       /lib/modules/6.8.0-45-generic/kernel/zfs/zfs.ko.zst
version:        2.2.2-0ubuntu9

I've also enabled SLUB debugging (in /etc/default/grub)

GRUB_CMDLINE_LINUX_DEFAULT="slub_debug=FZPU"

and KFENCE:

echo 1 > /sys/module/kfence/parameters/sample_interval
echo 10 > /sys/module/kfence/parameters/skip_covered_thresh

With no results, unfortunately. I also tried to limit physical memory amount from 256GiB to 128G, 64G and 32G with the same results. No crashes on docker run -it --rm quay.io/go-skynet/local-ai:latest-aio-gpu-hipblas.

Likely, it's a tricky race condition.

Oct 09 '24 14:10 mihalicyn

@mihalicyn maybe a silly question, but did you use Docker's zfs storage driver? On OpenZFS 2.2, it defaults to overlay2 IIRC

Oct 09 '24 14:10 andrebrait

@mihalicyn maybe a silly question, but did you use Docker's zfs storage driver? On OpenZFS 2.2, it defaults to overlay2 IIRC

not a silly question at all ;-) When debugging stuff everything must be checked twice!

Yeah, I do have ZFS storage driver enabled manually:

# docker info | grep zfs
 Storage Driver: zfs

# cat /proc/self/mountinfo | grep docker
48 31 0:36 / /var/lib/docker rw,relatime shared:46 - zfs zpool-docker rw,xattr,noacl,casesensitive

# zpool status
  pool: zpool-docker
 state: ONLINE
config:

	NAME          STATE     READ WRITE CKSUM
	zpool-docker  ONLINE       0     0     0
	  sdb         ONLINE       0     0     0

errors: No known data errors

Oct 09 '24 14:10 mihalicyn

That is very odd, it crashes every single time for me. It's a guaranteed crash whenever I run that command.

I'll see if I can make a VM that reproduces. In the meantime, I can provide any debug log asked.

Oct 09 '24 18:10 maxpoulin64

That is very odd, it crashes every single time for me. It's a guaranteed crash whenever I run that command.

Please can you try to enable SLAB debugging with slub_debug=FZPU kernel parameter and KFENCE (as I've described above)? This can help to identify where the issue is. KFENCE is safe to be enabled in production environment.

I'll see if I can make a VM that reproduces. In the meantime, I can provide any debug log asked.

That would be awesome too!

Oct 09 '24 18:10 mihalicyn

Alright, I've been working on this for a bit and I haven't been able to reproduce on my laptop, nor in a VM on said laptop. I then did a quick sanity check and yup, still crashes first time on my desktop.

So I decided to instrument the desktop, and then... nothing. It seems like slub_debug=FZPU slows the system too much, the extracting was much slower in Docker and it just never crashed. I wiped the whole Docker state and its dangling datasets, rebooted without the slub_debug=FZPU annnd it crashed on the first try. I'm also unable to reproduce with KFENCE enabled either, and I also notice with that one enabled that the extraction process is rather slow.

The trace I got this time however is different: it tried to execute an NX-protected page?

The only thing of note I can think of that might be a contributor is this is a Threadripper 1950X system, 16C/32T with 32 GB of RAM, so it's a NUMA system and relatively high core count, which leaves a lot of room for a race condition. Maybe if others in here can share their specs we can correlate some things.

[  270.176092] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[  270.176099] BUG: unable to handle page fault for address: ffff94c9c44fc400
[  270.176101] #PF: supervisor instruction fetch in kernel mode
[  270.176103] #PF: error_code(0x0011) - permissions violation
[  270.176106] PGD 5de601067 P4D 5de601067 PUD 1266f6063 PMD 172372063 PTE 80000001c44fc163
[  270.176112] Oops: Oops: 0011 [#1] PREEMPT SMP NOPTI
[  270.176116] CPU: 18 PID: 469 Comm: arc_prune Tainted: P        W  OE      6.10.10-zen1-1-zen #1 bb2e27e975e263b999d5cf1514b44f0d982487fe
[  270.176119] Hardware name: Micro-Star International Co., Ltd. MS-7B09/X399 GAMING PRO CARBON AC (MS-7B09), BIOS 1.B0 08/09/2018
[  270.176122] RIP: 0010:0xffff94c9c44fc400
[  270.176155] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 80 3e 45 00 c9 94 ff ff 00 00 00 00 01 01
[  270.176158] RSP: 0018:fffface8d0517cd8 EFLAGS: 00010246
[  270.176161] RAX: ffff94c9c44fc400 RBX: ffff94c96940e000 RCX: 0000000000000000
[  270.176163] RDX: 0000000000000000 RSI: fffface8d0517d20 RDI: ffff94c95b798000
[  270.176166] RBP: fffface8d0517d84 R08: 0000000000000000 R09: 0000000000000000
[  270.176168] R10: 0000000000000012 R11: 0000000000000001 R12: 00000000000007a3
[  270.176170] R13: ffff94c95b798000 R14: ffff94c96940e0f8 R15: ffff94cc842d0628
[  270.176172] FS:  0000000000000000(0000) GS:ffff94d06f700000(0000) knlGS:0000000000000000
[  270.176174] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  270.176177] CR2: ffff94c9c44fc400 CR3: 000000055dc9c000 CR4: 00000000003506f0
[  270.176179] Call Trace:
[  270.176182]  <TASK>
[  270.176184]  ? __die_body.cold+0x8/0x12
[  270.176190]  ? page_fault_oops+0x15a/0x2d0
[  270.176196]  ? exc_page_fault+0x18a/0x190
[  270.176200]  ? asm_exc_page_fault+0x26/0x30
[  270.176209]  ? zfs_prune+0xa9/0x4d0 [zfs a183bb1cfd40f08b536c8d802e186e2d90463da3]
[  270.176341]  ? zpl_prune_sb+0x36/0x60 [zfs a183bb1cfd40f08b536c8d802e186e2d90463da3]
[  270.176458]  ? arc_prune_task+0x22/0x40 [zfs a183bb1cfd40f08b536c8d802e186e2d90463da3]
[  270.176589]  ? taskq_thread+0x2e3/0x5d0 [spl 242ab66a9bc0261b2d10aafc27fd8fb6601368f0]
[  270.176600]  ? finish_task_switch.isra.0+0x99/0x2e0
[  270.176606]  ? __pfx_default_wake_function+0x10/0x10
[  270.176613]  ? __pfx_taskq_thread+0x10/0x10 [spl 242ab66a9bc0261b2d10aafc27fd8fb6601368f0]
[  270.176623]  ? kthread+0xd2/0x100
[  270.176627]  ? __pfx_kthread+0x10/0x10
[  270.176631]  ? ret_from_fork+0x34/0x50
[  270.176634]  ? __pfx_kthread+0x10/0x10
[  270.176637]  ? ret_from_fork_asm+0x1a/0x30
[  270.176644]  </TASK>
[  270.176646] Modules linked in: vhost_net vhost vhost_iotlb tap tun xt_conntrack nf_conntrack_netlink xfrm_user xfrm_algo ip6table_nat ip6table_filter ip6_tables xt_addrtype overlay rfcomm snd_seq_dummy snd_hrtimer snd_seq wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel bridge stp llc uhid cmac algif_hash algif_skcipher af_alg xt_MASQUERADE bnep xt_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_generic iptable_filter vfat fat intel_rapl_msr amd_atl intel_rapl_common iwlmvm snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul dm_crypt crc32_pclmul snd_hda_scodec_component snd_hda_codec_hdmi crc32c_intel btusb mac80211 polyval_clmulni snd_usb_audio btrtl snd_hda_intel polyval_generic btintel snd_intel_dspcfg gf128mul snd_usbmidi_lib ghash_clmulni_intel btbcm snd_intel_sdw_acpi snd_ump btmtk sha512_ssse3 libarc4 snd_hda_codec snd_rawmidi joydev sha256_ssse3 bluetooth snd_hda_core
[  270.176729]  snd_seq_device sha1_ssse3 iwlwifi mc snd_hwdep aesni_intel snd_pcm crypto_simd mousedev cryptd razerkbd(OE) crc16 wmi_bmof snd_timer rapl igb cbc cfg80211 snd ptp encrypted_keys pcspkr sp5100_tco mxm_wmi pps_core soundcore gpio_amdpt rfkill dca i2c_piix4 k10temp gpio_generic mac_hid kvmfr(OE) sg crypto_user loop nfnetlink ip_tables x_tables hid_logitech_hidpp hid_steam ff_memless hid_logitech_dj hid_generic trusted asn1_encoder tee dm_mod usbhid amdgpu vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd video amdxcp i2c_algo_bit drm_ttm_helper ttm drm_exec gpu_sched drm_suballoc_helper kvm_amd nvme drm_buddy drm_display_helper nvme_core xhci_pci kvm cec ccp nvme_auth xhci_pci_renesas wmi zfs(POE) spl(OE) vendor_reset(OE) nct6775 nct6775_core hwmon_vid i2c_dev
[  270.176812] CR2: ffff94c9c44fc400
[  270.176815] ---[ end trace 0000000000000000 ]---
[  270.176817] RIP: 0010:0xffff94c9c44fc400
[  270.176825] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 80 3e 45 00 c9 94 ff ff 00 00 00 00 01 01
[  270.176827] RSP: 0018:fffface8d0517cd8 EFLAGS: 00010246
[  270.176830] RAX: ffff94c9c44fc400 RBX: ffff94c96940e000 RCX: 0000000000000000
[  270.176832] RDX: 0000000000000000 RSI: fffface8d0517d20 RDI: ffff94c95b798000
[  270.176834] RBP: fffface8d0517d84 R08: 0000000000000000 R09: 0000000000000000
[  270.176836] R10: 0000000000000012 R11: 0000000000000001 R12: 00000000000007a3
[  270.176839] R13: ffff94c95b798000 R14: ffff94c96940e0f8 R15: ffff94cc842d0628
[  270.176841] FS:  0000000000000000(0000) GS:ffff94d06f700000(0000) knlGS:0000000000000000
[  270.176843] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  270.176845] CR2: ffff94c9c44fc400 CR3: 000000055dc9c000 CR4: 00000000003506f0
[  270.176848] note: arc_prune[469] exited with irqs disabled

Oct 10 '24 05:10 maxpoulin64

Additional information that I think could help narrow it down: I think it's possibly related to the creation and destruction of datasets and snapshots. I never see it die during the extraction, I see it die at the very end of it when it commits the Docker layer, whatever it's doing.

6ddbee975253: Extracting  352.2MB/352.2MB  <-- that's reporting 100% but never completes

That smells like a use after free race condition triggered by changes in datasets and snapshots. Is there an existing stress test for that I could try? I'll see if I can write one later this weekend when I have more time.

Oct 10 '24 05:10 maxpoulin64

Alright, I've been working on this for a bit and I haven't been able to reproduce on my laptop, nor in a VM on said laptop. I then did a quick sanity check and yup, still crashes first time on my desktop.

Thanks for doing that!

So I decided to instrument the desktop, and then... nothing. It seems like slub_debug=FZPU slows the system too much, the extracting was much slower in Docker and it just never crashed. I wiped the whole Docker state and its dangling datasets, rebooted without the slub_debug=FZPU annnd it crashed on the first try.

yeah, it can make things way too slow. You can play with parameters slub_debug and consider another combination, for example ZP can be very useful.

I'm also unable to reproduce with KFENCE enabled either, and I also notice with that one enabled that the extraction process is rather slow.

You can try to make kfence.sample_interval parameter to a bigger value instead of 1, let say 50 or 100. It make KFENCE precision worse but may help to get a reproduction of the issue.

Oct 10 '24 12:10 mihalicyn

I've observed the same error.

Context:

debian trixie
kernel 6.10.11-amd64
zfs 2.2.6-1
docker-ce 27.3.1
AMD Opteron(tm) X3421 APU

[Oct13 01:06] docker0: port 1(veth27f1644) entered disabled state
[  +0.000085] veth6b86ee8: renamed from eth0
[  +0.027785] docker0: port 1(veth27f1644) entered disabled state
[  +0.001856] veth27f1644 (unregistering): left allmulticast mode
[  +0.000010] veth27f1644 (unregistering): left promiscuous mode
[  +0.000025] docker0: port 1(veth27f1644) entered disabled state
[ +22.849012] BUG: unable to handle page fault for address: 0000000200000000
[  +0.000012] #PF: supervisor instruction fetch in kernel mode
[  +0.000003] #PF: error_code(0x0010) - not-present page
[  +0.000003] PGD 0 P4D 0 
[  +0.000004] Oops: Oops: 0010 [#1] PREEMPT SMP NOPTI
[  +0.000006] CPU: 3 PID: 983 Comm: arc_prune Tainted: P           OE      6.10.11-amd64 #1  Debian 6.10.11-1
[  +0.000004] Hardware name: HPE ProLiant MicroServer Gen10/ProLiant MicroServer Gen10, BIOS 5.12 02/19/2020
[  +0.000003] RIP: 0010:0x200000000
[  +0.000026] Code: Unable to access opcode bytes at 0x1ffffffd6.
[  +0.000002] RSP: 0018:ffffa84e80c7fce8 EFLAGS: 00010246
[  +0.000004] RAX: 0000000200000000 RBX: ffffa84e80c7fd9c RCX: 0000000000000000
[  +0.000002] RDX: 0000000000000000 RSI: ffffa84e80c7fd38 RDI: ffff995e5c8e0100
[  +0.000003] RBP: 0000000000000f23 R08: ffff995d13300000 R09: ffff995e025c8428
[  +0.000002] R10: 0000000000000001 R11: 0000000000000000 R12: ffff995e5c8e0100
[  +0.000002] R13: ffff995e02a1f4f0 R14: ffff995e025c8490 R15: ffff995d133000f8
[  +0.000002] FS:  0000000000000000(0000) GS:ffff9960ef580000(0000) knlGS:0000000000000000
[  +0.000003] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  +0.000002] CR2: 0000000200000000 CR3: 00000001719a6000 CR4: 00000000001506f0
[  +0.000003] Call Trace:
[  +0.000004]  <TASK>
[  +0.000003]  ? __die+0x23/0x70
[  +0.000008]  ? page_fault_oops+0x173/0x5a0
[  +0.000006]  ? exc_page_fault+0x7e/0x180
[  +0.000006]  ? asm_exc_page_fault+0x26/0x30
[  +0.000010]  ? zfs_prune+0xba/0x4e0 [zfs]
[  +0.000631]  ? finish_task_switch.isra.0+0x97/0x2c0
[  +0.000012]  ? zpl_prune_sb+0x38/0x60 [zfs]
[  +0.000579]  ? arc_prune_task+0x22/0x40 [zfs]
[  +0.000519]  ? taskq_thread+0x2ba/0x500 [spl]
[  +0.000025]  ? __pfx_default_wake_function+0x10/0x10
[  +0.000008]  ? __pfx_taskq_thread+0x10/0x10 [spl]
[  +0.000015]  ? kthread+0xd2/0x100
[  +0.000004]  ? __pfx_kthread+0x10/0x10
[  +0.000004]  ? ret_from_fork+0x34/0x50
[  +0.000004]  ? __pfx_kthread+0x10/0x10
[  +0.000003]  ? ret_from_fork_asm+0x1a/0x30
[  +0.000005]  </TASK>
[  +0.000002] Modules linked in: xt_nat xt_tcpudp veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat bridge stp llc nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtyp>
[  +0.000091]  crct10dif_common usbcore crc32c_intel scsi_mod libphy usb_common i2c_piix4 scsi_common video wmi
[  +0.000016] CR2: 0000000200000000
[  +0.000004] ---[ end trace 0000000000000000 ]---

Oct 14 '24 10:10 arcenik

The only thing of note I can think of that might be a contributor is this is a Threadripper 1950X system, 16C/32T with 32 GB of RAM, so it's a NUMA system and relatively high core count, which leaves a lot of room for a race condition. Maybe if others in here can share their specs we can correlate some things.

I've reproduced on every system I have in production, which are all "above average" core count systems. AMD Threadripper 3990X (64c/128t), Ryzen 3900X, 5900X (12c/24t). I can't try slub_debug or KFENCE in production, but I could try within KVM on the same systems.

Oct 17 '24 05:10 TheUbuntuGuy

The only thing of note I can think of that might be a contributor is this is a Threadripper 1950X system, 16C/32T with 32 GB of RAM, so it's a NUMA system and relatively high core count, which leaves a lot of room for a race condition. Maybe if others in here can share their specs we can correlate some things.

We also observed this across many high core count systems, in our case dual Xeon Platinum systems with 56C/112T

Oct 17 '24 06:10 webdock-io

Hi @TheUbuntuGuy,

I can't try slub_debug or KFENCE in production, but I could try within KVM on the same systems.

actually, you can enable KFENCE in production. It is designed to be enabled in production to debug issues like this one.

Oct 17 '24 07:10 mihalicyn

I reverted the kernel on my production systems to v6.6 as they were unusable due to this bug. I meant that I couldn't test using those options due to it being a production system and I can't crash it. I will try in a VM using the same CPU layout when I get the chance, probably with some automation to try the crash over and over, since KFENCE is sampled and may not catch the issue quickly.

Oct 18 '24 01:10 TheUbuntuGuy

I've started testing and this appears to be a use-after-free issue. With SLUB debugging enabled, I have triggered the bug multiple times, yielding:

general protection fault, probably for non-canonical address 0x6b6b6b6b6b6b6b6b: 0000 [#1] PREEMPT SMP NOPTI

In all cases, the address is the same, and 0x6b corresponds to the POISON_FREE value (from slub_debug=P).

A complete trace is:

Oct 18 00:08:04 zfstest kernel: general protection fault, probably for non-canonical address 0x6b6b6b6b6b6b6b6b: 0000 [#1] PREEMPT SMP NOPTI
Oct 18 00:08:04 zfstest kernel: CPU: 2 PID: 1478 Comm: arc_prune Tainted: P        W  O       6.8.9-xxx #1
Oct 18 00:08:04 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 00:08:04 zfstest kernel: RIP: 0010:__x86_indirect_thunk_array+0xa/0x20
Oct 18 00:08:04 zfstest kernel: Code: 57 fe ff ff eb e5 e8 15 0a 41 ff e9 86 fe ff ff 0f 0b e9 ea fe ff ff cc cc cc cc cc cc cc cc cc e8 01 00 00 00 cc 48 89 04 24 <c3> 66 66 2e 0f 1f 84 00 00 00 00 00 66 2e 0f 1f 84 00 00 00 00 00
Oct 18 00:08:04 zfstest kernel: RSP: 0018:ffffb3a5c480fd00 EFLAGS: 00010246
Oct 18 00:08:04 zfstest kernel: RAX: 6b6b6b6b6b6b6b6b RBX: ffff9836b99b2000 RCX: 0000000000000000
Oct 18 00:08:04 zfstest kernel: RDX: 0000000000000001 RSI: ffffb3a5c480fd58 RDI: ffff983699821880
Oct 18 00:08:04 zfstest kernel: RBP: ffffb3a5c480fdbc R08: 0000000000008fc9 R09: 0000000000000000
Oct 18 00:08:04 zfstest kernel: R10: ffff983516cc2ab0 R11: 0000000000000000 R12: 0000000000008fc9
Oct 18 00:08:04 zfstest kernel: R13: ffff983623f5b800 R14: ffff9836b99b20f8 R15: ffff983699821880
Oct 18 00:08:04 zfstest kernel: FS:  0000000000000000(0000) GS:ffff983775e80000(0000) knlGS:0000000000000000
Oct 18 00:08:04 zfstest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 18 00:08:04 zfstest kernel: CR2: 000000c022a0cd20 CR3: 0000000058c2e000 CR4: 0000000000350ef0
Oct 18 00:08:04 zfstest kernel: Call Trace:
Oct 18 00:08:04 zfstest kernel:  <TASK>
Oct 18 00:08:04 zfstest kernel:  ? __die_body+0x1b/0x60
Oct 18 00:08:04 zfstest kernel:  ? die_addr+0x39/0x60
Oct 18 00:08:04 zfstest kernel:  ? exc_general_protection+0x246/0x450
Oct 18 00:08:04 zfstest kernel:  ? asm_exc_general_protection+0x22/0x30
Oct 18 00:08:04 zfstest kernel:  ? __x86_indirect_thunk_array+0xa/0x20
Oct 18 00:08:04 zfstest kernel:  ? zfs_prune+0x9f/0x4a0 [zfs]
Oct 18 00:08:04 zfstest kernel:  ? zpl_prune_sb+0x34/0x50 [zfs]
Oct 18 00:08:04 zfstest kernel:  ? arc_prune_task+0x1e/0x30 [zfs]
Oct 18 00:08:04 zfstest kernel:  ? taskq_thread+0x271/0x470 [spl]
Oct 18 00:08:04 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 00:08:04 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 00:08:04 zfstest kernel:  ? kthread+0xf0/0x120
Oct 18 00:08:04 zfstest kernel:  ? kthread_complete_and_exit+0x20/0x20
Oct 18 00:08:04 zfstest kernel:  ? ret_from_fork+0x30/0x50
Oct 18 00:08:04 zfstest kernel:  ? kthread_complete_and_exit+0x20/0x20
Oct 18 00:08:04 zfstest kernel:  ? ret_from_fork_asm+0x11/0x20
Oct 18 00:08:04 zfstest kernel:  </TASK>
Oct 18 00:08:04 zfstest kernel: Modules linked in: nft_chain_nat xt_MASQUERADE nf_nat bridge stp llc nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables libcrc32c nfnetlink nvme_fabrics overlay binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common cfg80211 crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 snd_pcsp sha256_ssse3 qxl sha1_ssse3 snd_pcm input_leds drm_ttm_helper aesni_intel snd_timer ttm iTCO_wdt snd crypto_simd intel_pmc_bxt joydev serio_raw drm_kms_helper soundcore cryptd iTCO_vendor_support mac_hid drm efi_pstore dmi_sysfs qemu_fw_cfg virtio_rng ip_tables x_tables autofs4 zfs(PO) spl(O) hid_generic usbhid hid ahci i2c_i801 crc32_pclmul psmouse libahci lpc_ich xhci_pci i2c_smbus virtio_net virtio_blk xhci_pci_renesas
Oct 18 00:08:04 zfstest kernel: ---[ end trace 0000000000000000 ]---
Oct 18 00:08:05 zfstest kernel: RIP: 0010:__x86_indirect_thunk_array+0xa/0x20
Oct 18 00:08:05 zfstest kernel: Code: 57 fe ff ff eb e5 e8 15 0a 41 ff e9 86 fe ff ff 0f 0b e9 ea fe ff ff cc cc cc cc cc cc cc cc cc e8 01 00 00 00 cc 48 89 04 24 <c3> 66 66 2e 0f 1f 84 00 00 00 00 00 66 2e 0f 1f 84 00 00 00 00 00
Oct 18 00:08:05 zfstest kernel: RSP: 0018:ffffb3a5c480fd00 EFLAGS: 00010246
Oct 18 00:08:05 zfstest kernel: RAX: 6b6b6b6b6b6b6b6b RBX: ffff9836b99b2000 RCX: 0000000000000000
Oct 18 00:08:05 zfstest kernel: RDX: 0000000000000001 RSI: ffffb3a5c480fd58 RDI: ffff983699821880
Oct 18 00:08:05 zfstest kernel: RBP: ffffb3a5c480fdbc R08: 0000000000008fc9 R09: 0000000000000000
Oct 18 00:08:05 zfstest kernel: R10: ffff983516cc2ab0 R11: 0000000000000000 R12: 0000000000008fc9
Oct 18 00:08:05 zfstest kernel: R13: ffff983623f5b800 R14: ffff9836b99b20f8 R15: ffff983699821880
Oct 18 00:08:05 zfstest kernel: FS:  0000000000000000(0000) GS:ffff983776180000(0000) knlGS:0000000000000000
Oct 18 00:08:05 zfstest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 18 00:08:05 zfstest kernel: CR2: 000000c02a3ca1e0 CR3: 0000000120536000 CR4: 0000000000350ef0

I have managed to get a single reproduction with KFENCE enabled (71ms period), but no checks tripped. I will continue testing since I know it is possible.

Oct 18 '24 04:10 TheUbuntuGuy

I build a new kernel (6.10.14) with debug symbols so I could decode the stack traces, but ultimately forgot to set ZFS to also have debug enabled (doh!). Before I realized this, I got the following kernel bugs, which all appear to be from slab_debug=ZFOP. We are seeing the redzone get overwritten. I've collapsed the 2 traces since they are quite long.

Trace 1

Oct 18 20:08:37 zfstest kernel: =============================================================================
Oct 18 20:08:37 zfstest kernel: BUG dnode_t (Tainted: P           O      ): Left Redzone overwritten
Oct 18 20:08:37 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:37 zfstest kernel: 0xffff8d800aae8000-0xffff8d800aae8007 @offset=0. First byte 0xec instead of 0xcc
Oct 18 20:08:37 zfstest kernel: Slab 0xffffd668882aba00 objects=32 used=30 fp=0xffff8d800aaeabb0 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:37 zfstest kernel: Object 0xffff8d800aae8008 @offset=8 fp=0xffff8d800aaed360
Oct 18 20:08:37 zfstest kernel: Redzone  ffff8d800aae8000: ec f3 9e 56 c7 b0 24 32                          ...V..$2
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8008: 89 e4 3e e8 98 16 f5 be c1 33 51 15 7f 8d ff ff  ..>......3Q.....
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8018: 00 00 00 00 00 00 00 00 20 80 ae 0a 80 8d ff ff  ........ .......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8028: 20 80 ae 0a 80 8d ff ff 00 00 00 00 00 00 00 00   ...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8038: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8048: 01 50 5c 96 7f 8d ff ff b4 db 00 00 00 00 00 00  .P\.............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8058: 68 ae f5 ad 7f 8d ff ff 40 6b f9 e0 7f 8d ff ff  h.......@k......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8068: 00 e8 fd 6e 80 8d ff ff 13 00 00 00 b0 00 2c 01  ...n..........,.
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8078: 00 00 01 11 00 00 0a 00 00 14 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8088: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8098: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80a8: 00 6b 00 00 00 00 00 00 00 00 6b 6b 00 00 00 00  .k........kk....
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80b8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80e8: 00 00 00 00 6b 6b 6b 6b 00 01 00 00 00 00 ad de  ....kkkk........
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80f8: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8108: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8118: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8128: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8138: 00 00 00 00 00 00 00 00 40 81 ae 0a 80 8d ff ff  ........@.......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8148: 40 81 ae 0a 80 8d ff ff 00 00 00 00 6b 6b 6b 6b  @...........kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8158: 00 00 00 00 00 00 00 00 98 01 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8168: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8178: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8188: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8198: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81a8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81b8: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81c8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81d8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8208: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8218: 00 00 00 00 00 00 00 00 f5 45 65 34 6b 6b 6b 6b  .........Ee4kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8228: 00 00 00 00 6b 6b 6b 6b 30 82 ae 0a 80 8d ff ff  ....kkkk0.......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8238: 30 82 ae 0a 80 8d ff ff 00 00 00 00 6b 6b 6b 6b  0...........kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8248: 48 82 ae 0a 80 8d ff ff 48 82 ae 0a 80 8d ff ff  H.......H.......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8258: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8268: f5 45 65 34 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b  .Ee4kkkk....kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8278: 78 82 ae 0a 80 8d ff ff 78 82 ae 0a 80 8d ff ff  x.......x.......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8288: 00 00 00 00 6b 6b 6b 6b 90 82 ae 0a 80 8d ff ff  ....kkkk........
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8298: 90 82 ae 0a 80 8d ff ff 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82a8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82b8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82d8: 00 00 00 00 00 00 00 00 e0 82 ae 0a 80 8d ff ff  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82e8: e0 82 ae 0a 80 8d ff ff 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8308: 20 b5 51 c0 ff ff ff ff 40 00 00 00 00 00 00 00   .Q.....@.......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8318: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8328: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8338: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8348: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8358: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8368: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8378: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8388: 00 00 00 00 00 00 00 00 90 83 ae 0a 80 8d ff ff  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8398: 90 83 ae 0a 80 8d ff ff 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae83a8: 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae83b8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae83c8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae83d8: 00 00 00 00 6b 6b 6b a5                          ....kkk.
Oct 18 20:08:37 zfstest kernel: Redzone  ffff8d800aae83e0: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:37 zfstest kernel: Padding  ffff8d800aae83f0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:08:37 zfstest kernel: CPU: 92 PID: 1503 Comm: dbu_evict Tainted: P           O       6.10.14-xxx #1
Oct 18 20:08:37 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:37 zfstest kernel: Call Trace:
Oct 18 20:08:37 zfstest kernel:  <TASK>
Oct 18 20:08:37 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:08:37 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:08:37 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:08:37 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:08:37 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:08:37 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:08:37 zfstest kernel:  kmem_cache_free+0x354/0x3f0
Oct 18 20:08:37 zfstest kernel:  spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:08:37 zfstest kernel:  dnode_destroy+0x1df/0x240 [zfs]
Oct 18 20:08:37 zfstest kernel:  dnode_buf_evict_async+0x75/0xd0 [zfs]
Oct 18 20:08:37 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:08:37 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:08:37 zfstest kernel:  ? dnode_create+0x1e0/0x1e0 [zfs]
Oct 18 20:08:37 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:08:37 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:08:37 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:37 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:08:37 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:37 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:08:37 zfstest kernel:  </TASK>
Oct 18 20:08:37 zfstest kernel: FIX dnode_t: Restoring Left Redzone 0xffff8d800aae8000-0xffff8d800aae8007=0xcc
Oct 18 20:08:37 zfstest kernel: FIX dnode_t: Object at 0xffff8d800aae8008 not freed
Oct 18 20:08:38 zfstest kernel: =============================================================================
Oct 18 20:08:38 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:38 zfstest kernel: 0xffff8d7f417ae000-0xffff8d7f417ae00f @offset=0. First byte 0xb4 instead of 0xcc
Oct 18 20:08:38 zfstest kernel: Slab 0xffffd6688505eb80 objects=42 used=42 fp=0x0000000000000000 flags=0x17ffffc0000040(head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:38 zfstest kernel: Object 0xffff8d7f417ae040 @offset=64 fp=0x0000000000000000
Oct 18 20:08:38 zfstest kernel: Redzone  ffff8d7f417ae000: b4 d6 64 fb 00 bd 27 7c 1b f6 9b 8f 2a c3 eb 15  ..d...'|....*...
Oct 18 20:08:38 zfstest kernel: Redzone  ffff8d7f417ae010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:38 zfstest kernel: Redzone  ffff8d7f417ae020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:38 zfstest kernel: Redzone  ffff8d7f417ae030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:38 zfstest kernel: Object   ffff8d7f417ae040: 00 00 00 10 9f 01 1e 00 00 00 40 20 89 00 12 00  ..........@ ....
Oct 18 20:08:38 zfstest kernel: Object   ffff8d7f417ae050: 00 00 00 40 89 00 12 00 6b 6b 6b 6b 6b 6b 6b 6b  [email protected]
Oct 18 20:08:38 zfstest kernel: Object   ffff8d7f417ae060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:38 zfstest kernel: Object   ffff8d7f417ae070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:08:38 zfstest kernel: Redzone  ffff8d7f417ae080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:38 zfstest kernel: Padding  ffff8d7f417ae090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:38 zfstest kernel: Padding  ffff8d7f417ae0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:38 zfstest kernel: Padding  ffff8d7f417ae0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:38 zfstest kernel: CPU: 56 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:38 zfstest kernel: Call Trace:
Oct 18 20:08:38 zfstest kernel:  <TASK>
Oct 18 20:08:38 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:08:38 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:08:38 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:08:38 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:08:38 zfstest kernel:  ? zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:08:38 zfstest kernel:  ? zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:08:38 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:08:38 zfstest kernel:  zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:08:38 zfstest kernel:  zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:08:38 zfstest kernel:  zfs_acl_free+0xe/0x50 [zfs]
Oct 18 20:08:38 zfstest kernel:  zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:08:38 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:08:38 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:08:38 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:08:38 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:08:38 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:08:38 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:08:38 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:08:38 zfstest kernel:  zfs_prune+0x399/0x490 [zfs]
Oct 18 20:08:38 zfstest kernel:  zpl_prune_sb+0x34/0x50 [zfs]
Oct 18 20:08:38 zfstest kernel:  arc_prune_task+0x1b/0x30 [zfs]
Oct 18 20:08:38 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:08:38 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:08:38 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:08:38 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:08:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:38 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:08:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:38 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:08:38 zfstest kernel:  </TASK>
Oct 18 20:08:38 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff8d7f417ae000-0xffff8d7f417ae00f=0xcc
Oct 18 20:08:38 zfstest kernel: FIX kmalloc-64: Object at 0xffff8d7f417ae040 not freed
Oct 18 20:08:41 zfstest kernel: =============================================================================
Oct 18 20:08:41 zfstest kernel: BUG kmalloc-96 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:41 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:41 zfstest kernel: 0xffff8d7ffec02000-0xffff8d7ffec02007 @offset=0. First byte 0x88 instead of 0xcc
Oct 18 20:08:41 zfstest kernel: Slab 0xffffd66887fb0080 objects=64 used=43 fp=0xffff8d7ffec03688 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:41 zfstest kernel: Object 0xffff8d7ffec02008 @offset=8 fp=0xffff8d7ffec02d88
Oct 18 20:08:41 zfstest kernel: Redzone  ffff8d7ffec02000: 88 ae 6b 01 0a 5e 23 a3                          ..k..^#.
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02008: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02018: 01 00 00 00 00 00 00 00 d8 79 fe 73 7f 8d ff ff  .........y.s....
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02028: 00 00 00 00 00 00 00 00 00 41 5a 4c 80 8d ff ff  .........AZL....
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02038: 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  @...............
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02048: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02058: c0 f7 89 c0 ff ff ff ff 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:08:41 zfstest kernel: Redzone  ffff8d7ffec02068: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:41 zfstest kernel: Padding  ffff8d7ffec02078: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:08:41 zfstest kernel: CPU: 65 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:41 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:41 zfstest kernel: Call Trace:
Oct 18 20:08:41 zfstest kernel:  <TASK>
Oct 18 20:08:41 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:08:41 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:08:41 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:08:41 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:08:41 zfstest kernel:  ? zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:08:41 zfstest kernel:  ? zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:08:41 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:08:41 zfstest kernel:  ? zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:08:41 zfstest kernel:  zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:08:41 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:08:41 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:08:41 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:08:41 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:08:41 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:08:41 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:08:41 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:08:41 zfstest kernel:  zfs_prune+0x399/0x490 [zfs]
Oct 18 20:08:41 zfstest kernel:  zpl_prune_sb+0x34/0x50 [zfs]
Oct 18 20:08:41 zfstest kernel:  arc_prune_task+0x1b/0x30 [zfs]
Oct 18 20:08:41 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:08:41 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:08:41 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:08:41 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:08:41 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:41 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:08:41 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:41 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:08:41 zfstest kernel:  </TASK>
Oct 18 20:08:41 zfstest kernel: FIX kmalloc-96: Restoring Left Redzone 0xffff8d7ffec02000-0xffff8d7ffec02007=0xcc
Oct 18 20:08:41 zfstest kernel: FIX kmalloc-96: Object at 0xffff8d7ffec02008 not freed
Oct 18 20:08:42 zfstest kernel: =============================================================================
Oct 18 20:08:42 zfstest kernel: BUG kmalloc-512 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:42 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:42 zfstest kernel: 0xffff8d805d128000-0xffff8d805d12800f @offset=0. First byte 0xf9 instead of 0xcc
Oct 18 20:08:42 zfstest kernel: Slab 0xffffd66889744a00 objects=21 used=18 fp=0xffff8d805d12c400 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:42 zfstest kernel: Object 0xffff8d805d128200 @offset=512 fp=0x0000000000000000
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128000: f9 f4 6d b7 29 cb 6b da 48 b1 e5 98 ec 7f 75 e3  ..m.).k.H.....u.
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128040: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128050: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128060: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128070: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128080: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128090: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128100: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128110: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128120: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128130: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128140: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128150: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128160: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128170: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128180: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128190: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128200: 5a 50 2f 00 02 04 18 00 a4 81 00 00 00 00 00 00  ZP/.............
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128210: 40 d9 00 00 00 00 00 00 d9 05 00 00 00 00 00 00  @...............
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128230: 9a 03 00 00 00 00 00 00 04 00 00 00 08 84 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128240: 11 ed 8e 63 00 00 00 00 00 00 00 00 00 00 00 00  ...c............
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128250: 11 ed 8e 63 00 00 00 00 00 00 00 00 00 00 00 00  ...c............
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128260: 69 f8 12 67 00 00 00 00 00 3b 58 08 00 00 00 00  i..g.....;X.....
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128270: 69 f8 12 67 00 00 00 00 00 3b 58 08 00 00 00 00  i..g.....;X.....
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128280: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128290: 03 00 00 00 00 00 00 00 00 00 00 10 9f 01 1e 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282a0: 00 00 40 20 89 00 12 00 00 00 00 40 89 00 12 00  ..@ .......@....
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128340: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128350: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128360: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128370: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128380: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128390: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128400: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128410: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128420: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128430: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128440: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128450: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128460: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128470: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128480: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128490: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128500: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128510: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128520: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128530: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128540: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128550: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128560: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128570: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128580: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128590: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: CPU: 103 PID: 1503 Comm: dbu_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:42 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:42 zfstest kernel: Call Trace:
Oct 18 20:08:42 zfstest kernel:  <TASK>
Oct 18 20:08:42 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:08:42 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:08:42 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:08:42 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:08:42 zfstest kernel:  ? dbuf_destroy+0x31e/0x440 [zfs]
Oct 18 20:08:42 zfstest kernel:  ? dbuf_destroy+0x31e/0x440 [zfs]
Oct 18 20:08:42 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:08:42 zfstest kernel:  dbuf_destroy+0x31e/0x440 [zfs]
Oct 18 20:08:42 zfstest kernel:  dnode_destroy+0x142/0x240 [zfs]
Oct 18 20:08:42 zfstest kernel:  dnode_buf_evict_async+0x75/0xd0 [zfs]
Oct 18 20:08:42 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:08:42 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:08:42 zfstest kernel:  ? dnode_create+0x1e0/0x1e0 [zfs]
Oct 18 20:08:42 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:08:42 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:08:42 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:42 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:08:42 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:42 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:08:42 zfstest kernel:  </TASK>
Oct 18 20:08:42 zfstest kernel: FIX kmalloc-512: Restoring Left Redzone 0xffff8d805d128000-0xffff8d805d12800f=0xcc
Oct 18 20:08:42 zfstest kernel: FIX kmalloc-512: Object at 0xffff8d805d128200 not freed
Oct 18 20:08:43 zfstest kernel: =============================================================================
Oct 18 20:08:43 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:43 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:43 zfstest kernel: 0xffff8d7f9e5fa000-0xffff8d7f9e5fa00f @offset=0. First byte 0x49 instead of 0xcc
Oct 18 20:08:43 zfstest kernel: Slab 0xffffd66886797e80 objects=42 used=30 fp=0xffff8d7f9e5fa100 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:43 zfstest kernel: Object 0xffff8d7f9e5fa040 @offset=64 fp=0xffff8d7f9e5fbc00
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7f9e5fa000: 49 dc 09 93 87 31 d2 93 0e 0d dd 6d 72 61 72 36  I....1.....mrar6
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7f9e5fa010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7f9e5fa020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7f9e5fa030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7f9e5fa040: 00 00 00 10 9f 01 1e 00 00 00 40 20 88 00 12 00  ..........@ ....
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7f9e5fa050: 00 00 00 40 88 00 12 00 6b 6b 6b 6b 6b 6b 6b 6b  [email protected]
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7f9e5fa060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7f9e5fa070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7f9e5fa080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:43 zfstest kernel: Padding  ffff8d7f9e5fa090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:43 zfstest kernel: Padding  ffff8d7f9e5fa0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:43 zfstest kernel: Padding  ffff8d7f9e5fa0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:43 zfstest kernel: CPU: 23 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:43 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:43 zfstest kernel: Call Trace:
Oct 18 20:08:43 zfstest kernel:  <TASK>
Oct 18 20:08:43 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:08:43 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:08:43 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:08:43 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:08:43 zfstest kernel:  ? zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:08:43 zfstest kernel:  ? zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:08:43 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:08:43 zfstest kernel:  zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:08:43 zfstest kernel:  zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:08:43 zfstest kernel:  zfs_acl_free+0xe/0x50 [zfs]
Oct 18 20:08:43 zfstest kernel:  zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:08:43 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:08:43 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:08:43 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:08:43 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:08:43 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:08:43 zfstest kernel:  zfs_prune+0x399/0x490 [zfs]
Oct 18 20:08:43 zfstest kernel:  zpl_prune_sb+0x34/0x50 [zfs]
Oct 18 20:08:43 zfstest kernel:  arc_prune_task+0x1b/0x30 [zfs]
Oct 18 20:08:43 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:08:43 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:08:43 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:08:43 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:08:43 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:43 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:08:43 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:43 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:08:43 zfstest kernel:  </TASK>
Oct 18 20:08:43 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff8d7f9e5fa000-0xffff8d7f9e5fa00f=0xcc
Oct 18 20:08:43 zfstest kernel: FIX kmalloc-64: Object at 0xffff8d7f9e5fa040 not freed
Oct 18 20:08:43 zfstest kernel: =============================================================================
Oct 18 20:08:43 zfstest kernel: BUG dnode_t (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:43 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:43 zfstest kernel: 0xffff8d7e74400000-0xffff8d7e74400007 @offset=0. First byte 0x3e instead of 0xcc
Oct 18 20:08:43 zfstest kernel: Slab 0xffffd66881d10000 objects=32 used=32 fp=0x0000000000000000 flags=0xfffffc0000040(head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:08:43 zfstest kernel: Object 0xffff8d7e74400008 @offset=8 fp=0xffff8d7e74405360
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7e74400000: 3e 88 19 c6 ac f6 55 34                          >.....U4
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400008: 49 55 13 df 22 f5 59 cc c1 b3 6b fc 7f 8d ff ff  IU..".Y...k.....
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400018: 00 00 00 00 00 00 00 00 20 00 40 74 7e 8d ff ff  ........ .@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400028: 20 00 40 74 7e 8d ff ff 00 00 00 00 00 00 00 00   .@t~...........
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400038: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400048: 01 b8 c6 a1 7f 8d ff ff 02 f7 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400058: 98 50 aa 62 7e 8d ff ff b0 c1 88 47 7e 8d ff ff  .P.b~......G~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400068: 00 04 a0 3d 80 8d ff ff 14 00 00 00 b0 00 2c 01  ...=..........,.
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400078: 00 00 01 11 09 00 01 00 00 02 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400088: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400098: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000a8: 00 6b 00 00 00 00 00 00 00 00 6b 6b 00 00 00 00  .k........kk....
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000b8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000e8: 00 00 00 00 6b 6b 6b 6b 00 01 00 00 00 00 ad de  ....kkkk........
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000f8: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400108: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400118: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400128: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400138: 00 00 00 00 00 00 00 00 40 01 40 74 7e 8d ff ff  ........@.@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400148: 40 01 40 74 7e 8d ff ff 00 00 00 00 6b 6b 6b 6b  @.@t~.......kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400158: 00 00 00 00 00 00 00 00 98 01 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400168: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400178: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400188: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400198: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001a8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001b8: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001c8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001d8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400208: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400218: 00 00 00 00 00 00 00 00 f5 45 65 34 6b 6b 6b 6b  .........Ee4kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400228: 00 00 00 00 6b 6b 6b 6b 30 02 40 74 7e 8d ff ff  ....kkkk0.@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400238: 30 02 40 74 7e 8d ff ff 00 00 00 00 6b 6b 6b 6b  0.@t~.......kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400248: 48 02 40 74 7e 8d ff ff 48 02 40 74 7e 8d ff ff  H.@t~...H.@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400258: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400268: f5 45 65 34 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b  .Ee4kkkk....kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400278: 78 02 40 74 7e 8d ff ff 78 02 40 74 7e 8d ff ff  x.@t~...x.@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400288: 00 00 00 00 6b 6b 6b 6b 90 02 40 74 7e 8d ff ff  ....kkkk..@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400298: 90 02 40 74 7e 8d ff ff 00 00 00 00 00 00 00 00  ..@t~...........
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002a8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002b8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002d8: 00 00 00 00 00 00 00 00 e0 02 40 74 7e 8d ff ff  ..........@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002e8: e0 02 40 74 7e 8d ff ff 00 00 00 00 6b 6b 6b 6b  ..@t~.......kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400308: 20 b5 51 c0 ff ff ff ff 40 00 00 00 00 00 00 00   .Q.....@.......
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400318: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400328: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400338: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400348: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400358: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400368: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400378: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400388: 00 00 00 00 00 00 00 00 90 03 40 74 7e 8d ff ff  ..........@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400398: 90 03 40 74 7e 8d ff ff 00 00 00 00 6b 6b 6b 6b  ..@t~.......kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744003a8: 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744003b8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744003c8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744003d8: 00 00 00 00 6b 6b 6b a5                          ....kkk.
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7e744003e0: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:43 zfstest kernel: Padding  ffff8d7e744003f0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:08:43 zfstest kernel: CPU: 109 PID: 1503 Comm: dbu_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:43 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:43 zfstest kernel: Call Trace:
Oct 18 20:08:43 zfstest kernel:  <TASK>
Oct 18 20:08:43 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:08:43 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:08:43 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:08:43 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:08:43 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:08:43 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:08:43 zfstest kernel:  kmem_cache_free+0x354/0x3f0
Oct 18 20:08:43 zfstest kernel:  spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:08:43 zfstest kernel:  dnode_destroy+0x1df/0x240 [zfs]
Oct 18 20:08:43 zfstest kernel:  dnode_buf_evict_async+0x75/0xd0 [zfs]
Oct 18 20:08:43 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:08:43 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:08:43 zfstest kernel:  ? dnode_create+0x1e0/0x1e0 [zfs]
Oct 18 20:08:43 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:08:43 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:08:43 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:43 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:08:43 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:43 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:08:43 zfstest kernel:  </TASK>
Oct 18 20:08:43 zfstest kernel: FIX dnode_t: Restoring Left Redzone 0xffff8d7e74400000-0xffff8d7e74400007=0xcc
Oct 18 20:08:43 zfstest kernel: FIX dnode_t: Object at 0xffff8d7e74400008 not freed
Oct 18 20:08:44 zfstest kernel: =============================================================================
Oct 18 20:08:44 zfstest kernel: BUG kmalloc-32 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:44 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:44 zfstest kernel: 0xffff8d7ffb851000-0xffff8d7ffb85100f @offset=0. First byte 0x31 instead of 0xcc
Oct 18 20:08:44 zfstest kernel: Slab 0xffffd66887ee1440 objects=42 used=36 fp=0xffff8d7ffb851500 flags=0x17ffffc0000200(workingset|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:44 zfstest kernel: Object 0xffff8d7ffb851020 @offset=32 fp=0x0000000000000000
Oct 18 20:08:44 zfstest kernel: Redzone  ffff8d7ffb851000: 31 08 21 24 54 78 da 0d 4c b0 f1 fd b6 a4 ea b3  1.!$Tx..L.......
Oct 18 20:08:44 zfstest kernel: Redzone  ffff8d7ffb851010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:44 zfstest kernel: Object   ffff8d7ffb851020: 00 00 00 00 01 00 00 00 80 b4 f0 bd 7f 8d ff ff  ................
Oct 18 20:08:44 zfstest kernel: Object   ffff8d7ffb851030: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:08:44 zfstest kernel: Redzone  ffff8d7ffb851040: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:44 zfstest kernel: Padding  ffff8d7ffb851050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:44 zfstest kernel: CPU: 23 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:44 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:44 zfstest kernel: Call Trace:
Oct 18 20:08:44 zfstest kernel:  <TASK>
Oct 18 20:08:44 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:08:44 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:08:44 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:08:44 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:08:44 zfstest kernel:  ? nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:08:44 zfstest kernel:  ? nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:08:44 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:08:44 zfstest kernel:  ? _raw_spin_unlock_irqrestore+0x23/0x40
Oct 18 20:08:44 zfstest kernel:  nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:08:44 zfstest kernel:  zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:08:44 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:08:44 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:08:44 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:08:44 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:08:44 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:08:44 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:08:44 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:08:44 zfstest kernel:  zfs_prune+0x399/0x490 [zfs]
Oct 18 20:08:44 zfstest kernel:  zpl_prune_sb+0x34/0x50 [zfs]
Oct 18 20:08:44 zfstest kernel:  arc_prune_task+0x1b/0x30 [zfs]
Oct 18 20:08:44 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:08:44 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:08:44 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:08:44 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:08:44 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:44 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:08:44 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:44 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:08:44 zfstest kernel:  </TASK>
Oct 18 20:08:44 zfstest kernel: FIX kmalloc-32: Restoring Left Redzone 0xffff8d7ffb851000-0xffff8d7ffb85100f=0xcc
Oct 18 20:08:44 zfstest kernel: FIX kmalloc-32: Object at 0xffff8d7ffb851020 not freed
Oct 18 20:08:46 zfstest kernel: =============================================================================
Oct 18 20:08:46 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:46 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:46 zfstest kernel: 0xffff8d806862c000-0xffff8d806862c00f @offset=0. First byte 0x5 instead of 0xcc
Oct 18 20:08:46 zfstest kernel: Slab 0xffffd66889a18b00 objects=42 used=24 fp=0xffff8d806862d600 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:46 zfstest kernel: Object 0xffff8d806862c040 @offset=64 fp=0x0000000000000000
Oct 18 20:08:46 zfstest kernel: Redzone  ffff8d806862c000: 05 85 17 77 57 a1 e4 98 9a f6 d0 7f 5b b9 1d f9  ...wW.......[...
Oct 18 20:08:46 zfstest kernel: Redzone  ffff8d806862c010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:46 zfstest kernel: Redzone  ffff8d806862c020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:46 zfstest kernel: Redzone  ffff8d806862c030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:46 zfstest kernel: Object   ffff8d806862c040: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:08:46 zfstest kernel: Object   ffff8d806862c050: 80 9a cf c4 7f 8d ff ff 80 9a cf c4 7f 8d ff ff  ................
Oct 18 20:08:46 zfstest kernel: Object   ffff8d806862c060: 30 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00  0...............
Oct 18 20:08:46 zfstest kernel: Object   ffff8d806862c070: 03 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00  ................
Oct 18 20:08:46 zfstest kernel: Redzone  ffff8d806862c080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:46 zfstest kernel: Padding  ffff8d806862c090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:46 zfstest kernel: Padding  ffff8d806862c0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:46 zfstest kernel: Padding  ffff8d806862c0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:46 zfstest kernel: CPU: 30 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:46 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:46 zfstest kernel: Call Trace:
Oct 18 20:08:46 zfstest kernel:  <TASK>
Oct 18 20:08:46 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:08:46 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:08:46 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:08:46 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:08:46 zfstest kernel:  ? zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:08:46 zfstest kernel:  ? zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:08:46 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:08:46 zfstest kernel:  zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:08:46 zfstest kernel:  zfs_acl_free+0xe/0x50 [zfs]
Oct 18 20:08:46 zfstest kernel:  zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:08:46 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:08:46 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:08:46 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:08:46 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:08:46 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:08:46 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:08:46 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:08:46 zfstest kernel:  zfs_prune+0x399/0x490 [zfs]
Oct 18 20:08:46 zfstest kernel:  zpl_prune_sb+0x34/0x50 [zfs]
Oct 18 20:08:46 zfstest kernel:  arc_prune_task+0x1b/0x30 [zfs]
Oct 18 20:08:46 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:08:46 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:08:46 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:08:46 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:08:46 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:46 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:08:46 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:46 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:08:46 zfstest kernel:  </TASK>
Oct 18 20:08:46 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff8d806862c000-0xffff8d806862c00f=0xcc
Oct 18 20:08:46 zfstest kernel: FIX kmalloc-64: Object at 0xffff8d806862c040 not freed
Oct 18 20:08:47 zfstest kernel: =============================================================================
Oct 18 20:08:47 zfstest kernel: BUG kmalloc-96 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:47 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:47 zfstest kernel: 0xffff8d7fd8982000-0xffff8d7fd8982007 @offset=0. First byte 0xfe instead of 0xcc
Oct 18 20:08:47 zfstest kernel: Slab 0xffffd66887626080 objects=64 used=47 fp=0xffff8d7fd8982f88 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:47 zfstest kernel: Object 0xffff8d7fd8982008 @offset=8 fp=0xffff8d7fd8982108
Oct 18 20:08:47 zfstest kernel: Redzone  ffff8d7fd8982000: fe a4 c1 99 23 d9 28 30                          ....#.(0
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982008: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982018: 01 00 00 00 00 00 00 00 18 2d cd 88 7f 8d ff ff  .........-......
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982028: 00 00 00 00 00 00 00 00 80 8e d9 3a 80 8d ff ff  ...........:....
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982038: 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  @...............
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982048: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982058: c0 f7 89 c0 ff ff ff ff 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:08:47 zfstest kernel: Redzone  ffff8d7fd8982068: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:47 zfstest kernel: Padding  ffff8d7fd8982078: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:08:47 zfstest kernel: CPU: 53 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:47 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:47 zfstest kernel: Call Trace:
Oct 18 20:08:47 zfstest kernel:  <TASK>
Oct 18 20:08:47 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:08:47 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:08:47 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:08:47 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:08:47 zfstest kernel:  ? zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:08:47 zfstest kernel:  ? zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:08:47 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:08:47 zfstest kernel:  ? zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:08:47 zfstest kernel:  zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:08:47 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:08:47 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:08:47 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:08:47 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:08:47 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:08:47 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:08:47 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:08:47 zfstest kernel:  zfs_prune+0x399/0x490 [zfs]
Oct 18 20:08:47 zfstest kernel:  zpl_prune_sb+0x34/0x50 [zfs]
Oct 18 20:08:47 zfstest kernel:  arc_prune_task+0x1b/0x30 [zfs]
Oct 18 20:08:47 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:08:47 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:08:47 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:08:47 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:08:47 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:47 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:08:47 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:47 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:08:47 zfstest kernel:  </TASK>
Oct 18 20:08:47 zfstest kernel: FIX kmalloc-96: Restoring Left Redzone 0xffff8d7fd8982000-0xffff8d7fd8982007=0xcc
Oct 18 20:08:47 zfstest kernel: FIX kmalloc-96: Object at 0xffff8d7fd8982008 not freed
Oct 18 20:08:51 zfstest kernel: =============================================================================
Oct 18 20:08:51 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:51 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:51 zfstest kernel: 0xffff8d7feee7a000-0xffff8d7feee7a00f @offset=0. First byte 0xea instead of 0xcc
Oct 18 20:08:51 zfstest kernel: Slab 0xffffd66887bb9e80 objects=42 used=34 fp=0xffff8d7feee7ae80 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:51 zfstest kernel: Object 0xffff8d7feee7a040 @offset=64 fp=0x0000000000000000
Oct 18 20:08:51 zfstest kernel: Redzone  ffff8d7feee7a000: ea 20 d5 50 0a 52 f5 a0 75 eb d7 05 32 81 a0 7c  . .P.R..u...2..|
Oct 18 20:08:51 zfstest kernel: Redzone  ffff8d7feee7a010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:51 zfstest kernel: Redzone  ffff8d7feee7a020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:51 zfstest kernel: Redzone  ffff8d7feee7a030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:51 zfstest kernel: Object   ffff8d7feee7a040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:51 zfstest kernel: Object   ffff8d7feee7a050: 00 00 00 00 00 00 00 00 50 a3 6a c0 ff ff ff ff  ........P.j.....
Oct 18 20:08:51 zfstest kernel: Object   ffff8d7feee7a060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:51 zfstest kernel: Object   ffff8d7feee7a070: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:08:51 zfstest kernel: Redzone  ffff8d7feee7a080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:51 zfstest kernel: Padding  ffff8d7feee7a090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:51 zfstest kernel: Padding  ffff8d7feee7a0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:51 zfstest kernel: Padding  ffff8d7feee7a0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:51 zfstest kernel: CPU: 109 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:51 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:51 zfstest kernel: Call Trace:
Oct 18 20:08:51 zfstest kernel:  <TASK>
Oct 18 20:08:51 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:08:51 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:08:51 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:08:51 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:08:51 zfstest kernel:  ? zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:08:51 zfstest kernel:  ? zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:08:51 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:08:51 zfstest kernel:  zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:08:51 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:08:51 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:08:51 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:08:51 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:08:51 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:08:51 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:08:51 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:08:51 zfstest kernel:  zfs_prune+0x399/0x490 [zfs]
Oct 18 20:08:51 zfstest kernel:  zpl_prune_sb+0x34/0x50 [zfs]
Oct 18 20:08:51 zfstest kernel:  arc_prune_task+0x1b/0x30 [zfs]
Oct 18 20:08:51 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:08:51 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:08:51 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:08:51 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:08:51 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:51 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:08:51 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:51 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:08:51 zfstest kernel:  </TASK>
Oct 18 20:08:51 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff8d7feee7a000-0xffff8d7feee7a00f=0xcc
Oct 18 20:08:51 zfstest kernel: FIX kmalloc-64: Object at 0xffff8d7feee7a040 not freed
Oct 18 20:08:52 zfstest kernel: =============================================================================
Oct 18 20:08:52 zfstest kernel: BUG lsm_inode_cache (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:52 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:52 zfstest kernel: 0xffff8d7e40122000-0xffff8d7e40122007 @offset=0. First byte 0x5f instead of 0xcc
Oct 18 20:08:52 zfstest kernel: Slab 0xffffd66881004880 objects=42 used=11 fp=0xffff8d7e40122d28 flags=0xfffffc0000200(workingset|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:08:52 zfstest kernel: Object 0xffff8d7e40122008 @offset=8 fp=0xffff8d7e40122cc8
Oct 18 20:08:52 zfstest kernel: Redzone  ffff8d7e40122000: 5f 33 02 f5 75 d7 04 e6                          _3..u...
Oct 18 20:08:52 zfstest kernel: Object   ffff8d7e40122008: 88 fa 5c 1c 7e 8d ff ff 00 00 00 00 00 00 00 00  ..\.~...........
Oct 18 20:08:52 zfstest kernel: Object   ffff8d7e40122018: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:52 zfstest kernel: Object   ffff8d7e40122028: 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:52 zfstest kernel: Object   ffff8d7e40122038: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:52 zfstest kernel: Redzone  ffff8d7e40122048: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:52 zfstest kernel: Padding  ffff8d7e40122058: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:08:52 zfstest kernel: CPU: 92 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:52 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:52 zfstest kernel: Call Trace:
Oct 18 20:08:52 zfstest kernel:  <IRQ>
Oct 18 20:08:52 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:08:52 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:08:52 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:08:52 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:08:52 zfstest kernel:  ? rcu_core+0x430/0x770
Oct 18 20:08:52 zfstest kernel:  ? rcu_core+0x430/0x770
Oct 18 20:08:52 zfstest kernel:  kmem_cache_free+0x354/0x3f0
Oct 18 20:08:52 zfstest kernel:  ? rcu_core+0x3d0/0x770
Oct 18 20:08:52 zfstest kernel:  rcu_core+0x430/0x770
Oct 18 20:08:52 zfstest kernel:  ? kvm_clock_read+0x14/0x30
Oct 18 20:08:52 zfstest kernel:  handle_softirqs+0xd6/0x2a0
Oct 18 20:08:52 zfstest kernel:  irq_exit_rcu+0x6f/0xa0
Oct 18 20:08:52 zfstest kernel:  sysvec_apic_timer_interrupt+0xa1/0xd0
Oct 18 20:08:52 zfstest kernel:  </IRQ>
Oct 18 20:08:52 zfstest kernel:  <TASK>
Oct 18 20:08:52 zfstest kernel:  asm_sysvec_apic_timer_interrupt+0x16/0x20
Oct 18 20:08:52 zfstest kernel: RIP: 0010:_raw_spin_unlock_irqrestore+0x19/0x40
Oct 18 20:08:52 zfstest kernel: Code: d8 0d 6c 85 c0 74 01 c3 e8 d4 cb 2a ff c3 0f 1f 00 0f 1f 44 00 00 c6 07 00 0f 1f 00 f7 c6 00 02 00 00 74 06 fb 0f 1f 44 00 00 <bf> 01 00 00 00 e8 ed 03 39 ff 65 8b 05 9e d8 0d 6c 85 c0 74 01 c3
Oct 18 20:08:52 zfstest kernel: RSP: 0018:ffffa049049cfac8 EFLAGS: 00000206
Oct 18 20:08:52 zfstest kernel: RAX: ffff8d7fe443c580 RBX: ffff8d7fe443c580 RCX: 7643159d32c7016e
Oct 18 20:08:52 zfstest kernel: RDX: ffff8d7fe443f6b8 RSI: 0000000000000246 RDI: ffff8d7f15fbe640
Oct 18 20:08:52 zfstest kernel: RBP: ffffa049049cfb20 R08: ffff8d7fe443c688 R09: 0000000000000000
Oct 18 20:08:52 zfstest kernel: R10: ffff8d7fe443c678 R11: 0000000000000000 R12: ffffd66887910f00
Oct 18 20:08:52 zfstest kernel: R13: ffffd66887910f00 R14: ffff8d7f20fce980 R15: ffffffffc03f7e2b
Oct 18 20:08:52 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:08:52 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:08:52 zfstest kernel:  kmem_cache_free+0x354/0x3f0
Oct 18 20:08:52 zfstest kernel:  ? _raw_spin_lock+0x13/0x30
Oct 18 20:08:52 zfstest kernel:  spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:08:52 zfstest kernel:  zfs_zinactive+0x8f/0xf0 [zfs]
Oct 18 20:08:52 zfstest kernel:  zfs_inactive+0x7f/0x200 [zfs]
Oct 18 20:08:52 zfstest kernel:  ? unmap_mapping_pages+0x6f/0x120
Oct 18 20:08:52 zfstest kernel:  zpl_evict_inode+0x41/0x60 [zfs]
Oct 18 20:08:52 zfstest kernel:  evict+0xfd/0x260
Oct 18 20:08:52 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:08:52 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:08:52 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:08:52 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:08:52 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:08:52 zfstest kernel:  zfs_prune+0x399/0x490 [zfs]
Oct 18 20:08:52 zfstest kernel:  zpl_prune_sb+0x34/0x50 [zfs]
Oct 18 20:08:52 zfstest kernel:  arc_prune_task+0x1b/0x30 [zfs]
Oct 18 20:08:52 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:08:52 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:08:52 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:08:52 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:08:52 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:52 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:08:52 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:08:52 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:08:52 zfstest kernel:  </TASK>
Oct 18 20:08:52 zfstest kernel: FIX lsm_inode_cache: Restoring Left Redzone 0xffff8d7e40122000-0xffff8d7e40122007=0xcc
Oct 18 20:08:52 zfstest kernel: FIX lsm_inode_cache: Object at 0xffff8d7e40122008 not freed

Trace 2

Oct 18 20:15:35 zfstest kernel: =============================================================================
Oct 18 20:15:35 zfstest kernel: BUG kmalloc-512 (Tainted: P           O      ): Left Redzone overwritten
Oct 18 20:15:35 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:35 zfstest kernel: 0xffff9bec2d48f000-0xffff9bec2d48f00f @offset=0. First byte 0xa0 instead of 0xcc
Oct 18 20:15:35 zfstest kernel: Slab 0xffffe87080b523c0 objects=2 used=2 fp=0x0000000000000000 flags=0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:35 zfstest kernel: Object 0xffff9bec2d48f200 @offset=512 fp=0xffff9bec2d48f800
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f000: a0 f8 e7 f7 68 19 9d bc 91 8a 10 da d5 b4 bb f8  ....h...........
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f040: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f050: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f060: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f070: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f080: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f090: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f100: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f110: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f120: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f130: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f140: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f150: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f160: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f170: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f180: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f190: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f200: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f210: 70 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00  p...............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f220: 48 4d c9 02 ec 9b ff ff 00 01 00 00 00 00 ad de  HM..............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f230: 22 01 00 00 00 00 ad de b8 a7 72 62 ec 9b ff ff  ".........rb....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f290: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2d0: 00 b2 35 30 ec 9b ff ff 00 00 00 00 00 00 00 00  ..50............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f390: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b  ........kkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f400: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f410: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f420: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f430: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f440: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f450: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f460: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f470: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f480: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f490: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f500: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f510: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f520: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f530: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f540: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f550: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f560: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f570: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f580: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f590: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: CPU: 82 PID: 1917 Comm: dp_sync_taskq Tainted: P           O       6.10.14-xxx #1
Oct 18 20:15:35 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:35 zfstest kernel: Call Trace:
Oct 18 20:15:35 zfstest kernel:  <TASK>
Oct 18 20:15:35 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:35 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:35 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:35 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:35 zfstest kernel:  ? dbuf_sync_leaf+0x4b1/0x640 [zfs]
Oct 18 20:15:35 zfstest kernel:  ? dbuf_sync_leaf+0x4b1/0x640 [zfs]
Oct 18 20:15:35 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:35 zfstest kernel:  ? zio_taskq_dispatch+0x60/0xa0 [zfs]
Oct 18 20:15:35 zfstest kernel:  ? zio_nowait+0xad/0x140 [zfs]
Oct 18 20:15:35 zfstest kernel:  dbuf_sync_leaf+0x4b1/0x640 [zfs]
Oct 18 20:15:35 zfstest kernel:  ? zpl_get_file_info+0x86/0x230 [zfs]
Oct 18 20:15:35 zfstest kernel:  ? preempt_count_add+0x61/0xa0
Oct 18 20:15:35 zfstest kernel:  ? _raw_spin_lock+0x13/0x30
Oct 18 20:15:35 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:35 zfstest kernel:  ? dmu_objset_userquota_get_ids+0x37a/0x490 [zfs]
Oct 18 20:15:35 zfstest kernel:  dbuf_sync_list+0xa6/0x110 [zfs]
Oct 18 20:15:35 zfstest kernel:  dnode_sync+0x352/0xa80 [zfs]
Oct 18 20:15:35 zfstest kernel:  ? free_to_partial_list+0x107/0x520
Oct 18 20:15:35 zfstest kernel:  ? task_done+0x89/0x90 [spl]
Oct 18 20:15:35 zfstest kernel:  ? task_done+0x89/0x90 [spl]
Oct 18 20:15:35 zfstest kernel:  ? _raw_spin_unlock_irqrestore+0x23/0x40
Oct 18 20:15:35 zfstest kernel:  ? preempt_count_add+0x61/0xa0
Oct 18 20:15:35 zfstest kernel:  sync_dnodes_task+0x72/0xa0 [zfs]
Oct 18 20:15:35 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:15:35 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:15:35 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:15:35 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:35 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:35 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:35 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:35 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:35 zfstest kernel:  </TASK>
Oct 18 20:15:35 zfstest kernel: FIX kmalloc-512: Restoring Left Redzone 0xffff9bec2d48f000-0xffff9bec2d48f00f=0xcc
Oct 18 20:15:35 zfstest kernel: FIX kmalloc-512: Object at 0xffff9bec2d48f200 not freed
Oct 18 20:15:35 zfstest kernel: =============================================================================
Oct 18 20:15:35 zfstest kernel: BUG kmalloc-512 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:35 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:35 zfstest kernel: 0xffff9bec3150f000-0xffff9bec3150f00f @offset=0. First byte 0xe8 instead of 0xcc
Oct 18 20:15:35 zfstest kernel: Slab 0xffffe87080c543c0 objects=2 used=2 fp=0x0000000000000000 flags=0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:35 zfstest kernel: Object 0xffff9bec3150f200 @offset=512 fp=0x0000000000000000
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f000: e8 4a 9e 9d 18 08 18 88 ab 25 b8 b1 f9 dd f3 fb  .J.......%......
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f040: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f050: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f060: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f070: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f080: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f090: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f100: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f110: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f120: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f130: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f140: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f150: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f160: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f170: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f180: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f190: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f210: 10 f2 50 31 ec 9b ff ff 10 f2 50 31 ec 9b ff ff  ..P1......P1....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f240: 19 00 00 00 00 00 00 00 48 f2 50 31 ec 9b ff ff  ........H.P1....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f250: 48 f2 50 31 ec 9b ff ff 00 00 00 00 00 00 00 00  H.P1............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f280: 00 00 00 00 00 00 00 00 70 bb 5e c0 ff ff ff ff  ........p.^.....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f290: 00 00 00 00 00 00 00 00 00 88 f9 85 ed 9b ff ff  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2a0: ff 82 01 00 00 00 00 00 f8 87 61 3b ec 9b ff ff  ..........a;....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2b0: a8 bb 16 59 ec 9b ff ff 00 00 00 00 00 00 00 00  ...Y............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2d0: d0 f2 50 31 ec 9b ff ff d0 f2 50 31 ec 9b ff ff  ..P1......P1....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2e0: 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2f0: bf 8b fe 6b 00 00 00 00 04 00 07 00 04 00 00 00  ...k............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f300: 90 b7 5e c0 ff ff ff ff c0 b7 5e c0 ff ff ff ff  ..^.......^.....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f310: 08 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f320: 3e 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  >...............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f340: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b  ........kkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f350: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f360: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f370: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f380: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f390: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f400: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f410: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f420: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f430: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f440: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f450: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f460: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f470: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f480: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f490: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f500: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f510: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f520: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f530: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f540: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f550: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f560: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f570: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f580: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f590: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: CPU: 86 PID: 2814 Comm: z_wr_iss Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:35 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:35 zfstest kernel: Call Trace:
Oct 18 20:15:35 zfstest kernel:  <TASK>
Oct 18 20:15:35 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:35 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:35 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:35 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:35 zfstest kernel:  ? dbuf_evict_user.isra.0+0x31/0x60 [zfs]
Oct 18 20:15:35 zfstest kernel:  ? dbuf_evict_user.isra.0+0x31/0x60 [zfs]
Oct 18 20:15:35 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:35 zfstest kernel:  dbuf_evict_user.isra.0+0x31/0x60 [zfs]
Oct 18 20:15:35 zfstest kernel:  dbuf_clear_data+0x15/0x50 [zfs]
Oct 18 20:15:35 zfstest kernel:  dbuf_destroy+0x48/0x440 [zfs]
Oct 18 20:15:35 zfstest kernel:  dbuf_write_done+0xd4/0x1b0 [zfs]
Oct 18 20:15:35 zfstest kernel:  arc_write_done+0xa1/0x4e0 [zfs]
Oct 18 20:15:35 zfstest kernel:  zio_done+0x3d4/0xfc0 [zfs]
Oct 18 20:15:35 zfstest kernel:  ? zio_ready+0x1ac/0x3a0 [zfs]
Oct 18 20:15:35 zfstest kernel:  zio_execute+0x8b/0xe0 [zfs]
Oct 18 20:15:35 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:15:35 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:15:35 zfstest kernel:  ? zio_reexecute+0x3c0/0x3c0 [zfs]
Oct 18 20:15:35 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:15:35 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:35 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:35 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:35 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:35 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:35 zfstest kernel:  </TASK>
Oct 18 20:15:35 zfstest kernel: FIX kmalloc-512: Restoring Left Redzone 0xffff9bec3150f000-0xffff9bec3150f00f=0xcc
Oct 18 20:15:35 zfstest kernel: FIX kmalloc-512: Object at 0xffff9bec3150f200 not freed
Oct 18 20:15:35 zfstest dockerd[2741]: time="2024-10-18T20:15:35.677731588-04:00" level=info msg="Download failed, retrying (1/5): local error: tls: bad record MAC"
Oct 18 20:15:36 zfstest kernel: =============================================================================
Oct 18 20:15:36 zfstest kernel: BUG kmalloc-96 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:36 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:36 zfstest kernel: 0xffff9bec159d6000-0xffff9bec159d6007 @offset=0. First byte 0xd instead of 0xcc
Oct 18 20:15:36 zfstest kernel: Slab 0xffffe87080567580 objects=64 used=64 fp=0x0000000000000000 flags=0xfffffc0000040(head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:36 zfstest kernel: Object 0xffff9bec159d6008 @offset=8 fp=0xffff9bec159d6508
Oct 18 20:15:36 zfstest kernel: Redzone  ffff9bec159d6000: 0d ee 5b 6c 6c 4c ae 9f                          ..[llL..
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6008: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6018: 01 00 00 00 00 00 00 00 98 c5 93 44 ee 9b ff ff  ...........D....
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6028: 00 00 00 00 00 00 00 00 c0 c4 93 44 ee 9b ff ff  ...........D....
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6038: 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  @...............
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6048: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6058: c0 c7 8c c0 ff ff ff ff 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:36 zfstest kernel: Redzone  ffff9bec159d6068: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:36 zfstest kernel: Padding  ffff9bec159d6078: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:36 zfstest kernel: CPU: 91 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:36 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:36 zfstest kernel: Call Trace:
Oct 18 20:15:36 zfstest kernel:  <TASK>
Oct 18 20:15:36 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:36 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:36 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:36 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:36 zfstest kernel:  ? zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:15:36 zfstest kernel:  ? zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:15:36 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:36 zfstest kernel:  ? zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:15:36 zfstest kernel:  zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:15:36 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:15:36 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:15:36 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:15:36 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:36 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:15:36 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:15:36 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:15:36 zfstest kernel:  do_shrink_slab+0x132/0x340
Oct 18 20:15:36 zfstest kernel:  shrink_slab+0x7b/0x3d0
Oct 18 20:15:36 zfstest kernel:  shrink_node+0x2e5/0xab0
Oct 18 20:15:36 zfstest kernel:  ? sched_clock_cpu+0x11/0x1a0
Oct 18 20:15:36 zfstest kernel:  balance_pgdat+0x376/0x8b0
Oct 18 20:15:36 zfstest kernel:  ? finish_task_switch+0x95/0x280
Oct 18 20:15:36 zfstest kernel:  ? __x64_sys_sched_getscheduler+0x80/0x80
Oct 18 20:15:36 zfstest kernel:  ? _raw_spin_lock_irqsave+0x24/0x50
Oct 18 20:15:36 zfstest kernel:  ? lock_timer_base+0x82/0xa0
Oct 18 20:15:36 zfstest kernel:  kswapd+0x1e0/0x350
Oct 18 20:15:36 zfstest kernel:  ? prepare_to_wait_event+0x110/0x110
Oct 18 20:15:36 zfstest kernel:  ? balance_pgdat+0x8b0/0x8b0
Oct 18 20:15:36 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:36 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:36 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:36 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:36 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:36 zfstest kernel:  </TASK>
Oct 18 20:15:36 zfstest kernel: FIX kmalloc-96: Restoring Left Redzone 0xffff9bec159d6000-0xffff9bec159d6007=0xcc
Oct 18 20:15:36 zfstest kernel: FIX kmalloc-96: Object at 0xffff9bec159d6008 not freed
Oct 18 20:15:37 zfstest kernel: =============================================================================
Oct 18 20:15:37 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:37 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:37 zfstest kernel: 0xffff9bec110d0000-0xffff9bec110d000f @offset=0. First byte 0x71 instead of 0xcc
Oct 18 20:15:37 zfstest kernel: Slab 0xffffe87080443400 objects=42 used=21 fp=0xffff9bec110d1c00 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:37 zfstest kernel: Object 0xffff9bec110d0040 @offset=64 fp=0xffff9bec110d1840
Oct 18 20:15:37 zfstest kernel: Redzone  ffff9bec110d0000: 71 07 cf 9d e8 c5 26 64 02 6b ad 05 28 b0 c7 82  q.....&d.k..(...
Oct 18 20:15:37 zfstest kernel: Redzone  ffff9bec110d0010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:37 zfstest kernel: Redzone  ffff9bec110d0020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:37 zfstest kernel: Redzone  ffff9bec110d0030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:37 zfstest kernel: Object   ffff9bec110d0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:37 zfstest kernel: Object   ffff9bec110d0050: 00 00 00 00 00 00 00 00 50 73 6d c0 ff ff ff ff  ........Psm.....
Oct 18 20:15:37 zfstest kernel: Object   ffff9bec110d0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:37 zfstest kernel: Object   ffff9bec110d0070: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:37 zfstest kernel: Redzone  ffff9bec110d0080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:37 zfstest kernel: Padding  ffff9bec110d0090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:37 zfstest kernel: Padding  ffff9bec110d00a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:37 zfstest kernel: Padding  ffff9bec110d00b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:37 zfstest kernel: CPU: 81 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:37 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:37 zfstest kernel: Call Trace:
Oct 18 20:15:37 zfstest kernel:  <TASK>
Oct 18 20:15:37 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:37 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:37 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:37 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:37 zfstest kernel:  ? zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:15:37 zfstest kernel:  ? zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:15:37 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:37 zfstest kernel:  zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:15:37 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:15:37 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:15:37 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:15:37 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:37 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:15:37 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:15:37 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:15:37 zfstest kernel:  do_shrink_slab+0x132/0x340
Oct 18 20:15:37 zfstest kernel:  shrink_slab+0x7b/0x3d0
Oct 18 20:15:37 zfstest kernel:  shrink_node+0x2e5/0xab0
Oct 18 20:15:37 zfstest kernel:  ? sched_clock_cpu+0x11/0x1a0
Oct 18 20:15:37 zfstest kernel:  balance_pgdat+0x376/0x8b0
Oct 18 20:15:37 zfstest kernel:  ? finish_task_switch+0x95/0x280
Oct 18 20:15:37 zfstest kernel:  ? __x64_sys_sched_getscheduler+0x80/0x80
Oct 18 20:15:37 zfstest kernel:  ? _raw_spin_lock_irqsave+0x24/0x50
Oct 18 20:15:37 zfstest kernel:  ? lock_timer_base+0x82/0xa0
Oct 18 20:15:37 zfstest kernel:  kswapd+0x1e0/0x350
Oct 18 20:15:37 zfstest kernel:  ? prepare_to_wait_event+0x110/0x110
Oct 18 20:15:37 zfstest kernel:  ? balance_pgdat+0x8b0/0x8b0
Oct 18 20:15:37 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:37 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:37 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:37 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:37 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:37 zfstest kernel:  </TASK>
Oct 18 20:15:37 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bec110d0000-0xffff9bec110d000f=0xcc
Oct 18 20:15:37 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bec110d0040 not freed
Oct 18 20:15:38 zfstest kernel: =============================================================================
Oct 18 20:15:38 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:38 zfstest kernel: 0xffff9bec6168a000-0xffff9bec6168a00f @offset=0. First byte 0xe3 instead of 0xcc
Oct 18 20:15:38 zfstest kernel: Slab 0xffffe8708185a280 objects=42 used=23 fp=0xffff9bec6168ab80 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:38 zfstest kernel: Object 0xffff9bec6168a040 @offset=64 fp=0xffff9bec6168bf00
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6168a000: e3 41 57 24 d8 38 c2 16 a6 6d f7 5d 4f 8d 5b bd  .AW$.8...m.]O.[.
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6168a010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6168a020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6168a030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6168a040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6168a050: 00 00 00 00 00 00 00 00 50 73 6d c0 ff ff ff ff  ........Psm.....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6168a060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6168a070: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6168a080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec6168a090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec6168a0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec6168a0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: CPU: 43 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:38 zfstest kernel: Call Trace:
Oct 18 20:15:38 zfstest kernel:  <TASK>
Oct 18 20:15:38 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:38 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:38 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:38 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:38 zfstest kernel:  ? zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:15:38 zfstest kernel:  ? zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:15:38 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:38 zfstest kernel:  zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:15:38 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:15:38 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:15:38 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:15:38 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:38 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:15:38 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:15:38 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:15:38 zfstest kernel:  do_shrink_slab+0x132/0x340
Oct 18 20:15:38 zfstest kernel:  shrink_slab+0x7b/0x3d0
Oct 18 20:15:38 zfstest kernel:  shrink_node+0x2e5/0xab0
Oct 18 20:15:38 zfstest kernel:  ? sched_clock_cpu+0x11/0x1a0
Oct 18 20:15:38 zfstest kernel:  balance_pgdat+0x376/0x8b0
Oct 18 20:15:38 zfstest kernel:  ? finish_task_switch+0x95/0x280
Oct 18 20:15:38 zfstest kernel:  ? __x64_sys_sched_getscheduler+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ? _raw_spin_lock_irqsave+0x24/0x50
Oct 18 20:15:38 zfstest kernel:  ? lock_timer_base+0x82/0xa0
Oct 18 20:15:38 zfstest kernel:  kswapd+0x1e0/0x350
Oct 18 20:15:38 zfstest kernel:  ? prepare_to_wait_event+0x110/0x110
Oct 18 20:15:38 zfstest kernel:  ? balance_pgdat+0x8b0/0x8b0
Oct 18 20:15:38 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:38 zfstest kernel:  </TASK>
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bec6168a000-0xffff9bec6168a00f=0xcc
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bec6168a040 not freed
Oct 18 20:15:38 zfstest kernel: =============================================================================
Oct 18 20:15:38 zfstest kernel: BUG kmalloc-32 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:38 zfstest kernel: 0xffff9bec0f041000-0xffff9bec0f04100f @offset=0. First byte 0x4 instead of 0xcc
Oct 18 20:15:38 zfstest kernel: Slab 0xffffe870803c1040 objects=42 used=18 fp=0xffff9bec0f041920 flags=0xfffffc0000200(workingset|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:38 zfstest kernel: Object 0xffff9bec0f041020 @offset=32 fp=0xffff9bec0f041860
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec0f041000: 04 d3 b2 0a eb 4b 06 1c 15 d9 ab ac 48 54 18 e6  .....K......HT..
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec0f041010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec0f041020: 00 00 00 00 01 00 00 00 40 72 5d 49 ee 9b ff ff  ........@r]I....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec0f041030: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec0f041040: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec0f041050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: CPU: 43 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:38 zfstest kernel: Call Trace:
Oct 18 20:15:38 zfstest kernel:  <TASK>
Oct 18 20:15:38 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:38 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:38 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:38 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:38 zfstest kernel:  ? nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:38 zfstest kernel:  ? nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:38 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:38 zfstest kernel:  ? _raw_spin_unlock_irqrestore+0x23/0x40
Oct 18 20:15:38 zfstest kernel:  nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:38 zfstest kernel:  zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:15:38 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:15:38 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:15:38 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:15:38 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:38 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:15:38 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:15:38 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:15:38 zfstest kernel:  do_shrink_slab+0x132/0x340
Oct 18 20:15:38 zfstest kernel:  shrink_slab+0x7b/0x3d0
Oct 18 20:15:38 zfstest kernel:  shrink_node+0x2e5/0xab0
Oct 18 20:15:38 zfstest kernel:  ? sched_clock_cpu+0x11/0x1a0
Oct 18 20:15:38 zfstest kernel:  balance_pgdat+0x376/0x8b0
Oct 18 20:15:38 zfstest kernel:  ? finish_task_switch+0x95/0x280
Oct 18 20:15:38 zfstest kernel:  ? __x64_sys_sched_getscheduler+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ? _raw_spin_lock_irqsave+0x24/0x50
Oct 18 20:15:38 zfstest kernel:  ? lock_timer_base+0x82/0xa0
Oct 18 20:15:38 zfstest kernel:  kswapd+0x1e0/0x350
Oct 18 20:15:38 zfstest kernel:  ? prepare_to_wait_event+0x110/0x110
Oct 18 20:15:38 zfstest kernel:  ? balance_pgdat+0x8b0/0x8b0
Oct 18 20:15:38 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:38 zfstest kernel:  </TASK>
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-32: Restoring Left Redzone 0xffff9bec0f041000-0xffff9bec0f04100f=0xcc
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-32: Object at 0xffff9bec0f041020 not freed
Oct 18 20:15:38 zfstest kernel: =============================================================================
Oct 18 20:15:38 zfstest kernel: BUG dmu_buf_impl_t (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:38 zfstest kernel: 0xffff9bec6ec04000-0xffff9bec6ec04007 @offset=0. First byte 0xd3 instead of 0xcc
Oct 18 20:15:38 zfstest kernel: Slab 0xffffe87081bb0100 objects=38 used=34 fp=0xffff9bec6ec05f80 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:38 zfstest kernel: Object 0xffff9bec6ec04008 @offset=8 fp=0xffff9bec6ec05c30
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6ec04000: d3 89 f5 03 7b 3a 03 dd                          ....{:..
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04008: 3f ff a1 3e aa 4c a2 9f ff ff ff ff ff ff ff ff  ?..>.L..........
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04018: 40 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  @...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04028: 00 d8 b0 54 ee 9b ff ff d8 8c 5a 03 ec 9b ff ff  ...T......Z.....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04038: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04048: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04058: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04068: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04078: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04088: 00 00 00 00 00 00 00 00 90 40 c0 6e ec 9b ff ff  [email protected]....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04098: 90 40 c0 6e ec 9b ff ff 00 00 00 00 00 00 00 00  [email protected]............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040a8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040b8: 00 00 00 00 00 00 00 00 c0 40 c0 6e ec 9b ff ff  [email protected]....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040c8: c0 40 c0 6e ec 9b ff ff 00 00 00 00 00 00 00 00  [email protected]............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040d8: 00 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040e8: 00 00 00 00 00 00 00 00 f5 45 65 34 00 00 00 00  .........Ee4....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040f8: 00 00 00 00 00 00 00 00 00 41 c0 6e ec 9b ff ff  .........A.n....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04108: 00 41 c0 6e ec 9b ff ff 00 00 00 00 00 00 00 00  .A.n............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04118: 18 41 c0 6e ec 9b ff ff 18 41 c0 6e ec 9b ff ff  .A.n.....A.n....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04128: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04138: 00 00 00 00 00 00 00 00 98 01 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04148: 28 00 00 00 00 00 00 00 50 41 c0 6e ec 9b ff ff  (.......PA.n....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04158: 50 41 c0 6e ec 9b ff ff 00 01 00 00 00 00 ad de  PA.n............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04168: 22 01 00 00 00 00 ad de ff ff ff ff 00 00 00 00  "...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04178: 51 68 d5 2d 52 61 8b e0 00 00 00 00 00 00 00 00  Qh.-Ra..........
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04188: 01 00 00 00 00 00 00 00                          ........
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6ec04190: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec6ec041a0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: CPU: 115 PID: 1498 Comm: dbu_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:38 zfstest kernel: Call Trace:
Oct 18 20:15:38 zfstest kernel:  <TASK>
Oct 18 20:15:38 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:38 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:38 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:38 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:38 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:38 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:38 zfstest kernel:  kmem_cache_free+0x354/0x3f0
Oct 18 20:15:38 zfstest kernel:  spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:38 zfstest kernel:  dbuf_destroy+0x2a2/0x440 [zfs]
Oct 18 20:15:38 zfstest kernel:  dnode_destroy+0x142/0x240 [zfs]
Oct 18 20:15:38 zfstest kernel:  dnode_buf_evict_async+0x75/0xd0 [zfs]
Oct 18 20:15:38 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:15:38 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:15:38 zfstest kernel:  ? dnode_create+0x1e0/0x1e0 [zfs]
Oct 18 20:15:38 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:15:38 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:38 zfstest kernel:  </TASK>
Oct 18 20:15:38 zfstest kernel: FIX dmu_buf_impl_t: Restoring Left Redzone 0xffff9bec6ec04000-0xffff9bec6ec04007=0xcc
Oct 18 20:15:38 zfstest kernel: FIX dmu_buf_impl_t: Object at 0xffff9bec6ec04008 not freed
Oct 18 20:15:38 zfstest kernel: =============================================================================
Oct 18 20:15:38 zfstest kernel: BUG dmu_buf_impl_t (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:38 zfstest kernel: 0xffff9bec07d00000-0xffff9bec07d00007 @offset=0. First byte 0x1c instead of 0xcc
Oct 18 20:15:38 zfstest kernel: Slab 0xffffe870801f4000 objects=38 used=31 fp=0xffff9bec07d00ba0 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:38 zfstest kernel: Object 0xffff9bec07d00008 @offset=8 fp=0xffff9bec07d00ef0
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec07d00000: 1c 3b 92 46 54 5e 38 d7                          .;.FT^8.
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00008: 63 13 5d 6f 71 ab 0c f9 ff ff ff ff ff ff ff ff  c.]oq...........
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00018: 40 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  @...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00028: 00 10 74 e6 ed 9b ff ff e8 ed 92 52 ee 9b ff ff  ..t........R....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00038: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00048: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00058: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00068: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00078: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00088: 00 00 00 00 00 00 00 00 90 00 d0 07 ec 9b ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00098: 90 00 d0 07 ec 9b ff ff 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000a8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000b8: 00 00 00 00 00 00 00 00 c0 00 d0 07 ec 9b ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000c8: c0 00 d0 07 ec 9b ff ff 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000d8: 00 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000e8: 00 00 00 00 00 00 00 00 f5 45 65 34 00 00 00 00  .........Ee4....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000f8: 00 00 00 00 00 00 00 00 00 01 d0 07 ec 9b ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00108: 00 01 d0 07 ec 9b ff ff 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00118: 18 01 d0 07 ec 9b ff ff 18 01 d0 07 ec 9b ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00128: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00138: 00 00 00 00 00 00 00 00 98 01 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00148: 28 00 00 00 00 00 00 00 50 01 d0 07 ec 9b ff ff  (.......P.......
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00158: 50 01 d0 07 ec 9b ff ff 00 01 00 00 00 00 ad de  P...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00168: 22 01 00 00 00 00 ad de ff ff ff ff 00 00 00 00  "...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00178: 97 b7 d3 64 f3 3f 8a 4e 00 00 00 00 00 00 00 00  ...d.?.N........
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00188: 01 00 00 00 00 00 00 00                          ........
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec07d00190: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec07d001a0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: CPU: 115 PID: 1498 Comm: dbu_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:38 zfstest kernel: Call Trace:
Oct 18 20:15:38 zfstest kernel:  <TASK>
Oct 18 20:15:38 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:38 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:38 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:38 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:38 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:38 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:38 zfstest kernel:  kmem_cache_free+0x354/0x3f0
Oct 18 20:15:38 zfstest kernel:  spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:38 zfstest kernel:  dbuf_destroy+0x2a2/0x440 [zfs]
Oct 18 20:15:38 zfstest kernel:  dnode_destroy+0x142/0x240 [zfs]
Oct 18 20:15:38 zfstest kernel:  dnode_buf_evict_async+0x75/0xd0 [zfs]
Oct 18 20:15:38 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:15:38 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:15:38 zfstest kernel:  ? dnode_create+0x1e0/0x1e0 [zfs]
Oct 18 20:15:38 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:15:38 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:38 zfstest kernel:  </TASK>
Oct 18 20:15:38 zfstest kernel: FIX dmu_buf_impl_t: Restoring Left Redzone 0xffff9bec07d00000-0xffff9bec07d00007=0xcc
Oct 18 20:15:38 zfstest kernel: FIX dmu_buf_impl_t: Object at 0xffff9bec07d00008 not freed
Oct 18 20:15:38 zfstest kernel: =============================================================================
Oct 18 20:15:38 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:38 zfstest kernel: 0xffff9bec1db38000-0xffff9bec1db3800f @offset=0. First byte 0x54 instead of 0xcc
Oct 18 20:15:38 zfstest kernel: Slab 0xffffe8708076ce00 objects=42 used=29 fp=0xffff9bec1db38580 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:38 zfstest kernel: Object 0xffff9bec1db38040 @offset=64 fp=0xffff9bec1db38580
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec1db38000: 54 75 ea fd a0 d0 a6 7c 21 f2 fe 43 c3 43 1c a6  Tu.....|!..C.C..
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec1db38010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec1db38020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec1db38030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec1db38040: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec1db38050: 80 85 b3 1d ec 9b ff ff 80 85 b3 1d ec 9b ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec1db38060: 30 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00  0...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec1db38070: 03 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec1db38080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec1db38090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec1db380a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec1db380b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: CPU: 99 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:38 zfstest kernel: Call Trace:
Oct 18 20:15:38 zfstest kernel:  <TASK>
Oct 18 20:15:38 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:38 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:38 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:38 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:38 zfstest kernel:  ? zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:15:38 zfstest kernel:  ? zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:15:38 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:38 zfstest kernel:  zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:15:38 zfstest kernel:  zfs_acl_free+0xe/0x50 [zfs]
Oct 18 20:15:38 zfstest kernel:  zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:15:38 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:15:38 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:15:38 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:15:38 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:38 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:15:38 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:15:38 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:15:38 zfstest kernel:  do_shrink_slab+0x132/0x340
Oct 18 20:15:38 zfstest kernel:  shrink_slab+0x7b/0x3d0
Oct 18 20:15:38 zfstest kernel:  shrink_node+0x2e5/0xab0
Oct 18 20:15:38 zfstest kernel:  ? sched_clock_cpu+0x11/0x1a0
Oct 18 20:15:38 zfstest kernel:  balance_pgdat+0x376/0x8b0
Oct 18 20:15:38 zfstest kernel:  ? finish_task_switch+0x95/0x280
Oct 18 20:15:38 zfstest kernel:  ? __x64_sys_sched_getscheduler+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ? _raw_spin_lock_irqsave+0x24/0x50
Oct 18 20:15:38 zfstest kernel:  ? lock_timer_base+0x82/0xa0
Oct 18 20:15:38 zfstest kernel:  kswapd+0x1e0/0x350
Oct 18 20:15:38 zfstest kernel:  ? prepare_to_wait_event+0x110/0x110
Oct 18 20:15:38 zfstest kernel:  ? balance_pgdat+0x8b0/0x8b0
Oct 18 20:15:38 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:38 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:38 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:38 zfstest kernel:  </TASK>
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bec1db38000-0xffff9bec1db3800f=0xcc
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bec1db38040 not freed
Oct 18 20:15:39 zfstest kernel: =============================================================================
Oct 18 20:15:39 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:39 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:39 zfstest kernel: 0xffff9bec75fb6001-0xffff9bec75fb600f @offset=1. First byte 0xf8 instead of 0xcc
Oct 18 20:15:39 zfstest kernel: Slab 0xffffe87081d7ed80 objects=42 used=21 fp=0xffff9bec75fb6580 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:39 zfstest kernel: Object 0xffff9bec75fb6040 @offset=64 fp=0xffff9bec75fb6580
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec75fb6000: cc f8 7d 52 37 56 51 b3 06 2e cf ca 8e 96 ea b7  ..}R7VQ.........
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec75fb6010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec75fb6020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec75fb6030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec75fb6040: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec75fb6050: 80 65 fb 75 ec 9b ff ff 80 65 fb 75 ec 9b ff ff  .e.u.....e.u....
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec75fb6060: 30 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00  0...............
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec75fb6070: 03 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00  ................
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec75fb6080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:39 zfstest kernel: Padding  ffff9bec75fb6090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:39 zfstest kernel: Padding  ffff9bec75fb60a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:39 zfstest kernel: Padding  ffff9bec75fb60b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:39 zfstest kernel: CPU: 88 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:39 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:39 zfstest kernel: Call Trace:
Oct 18 20:15:39 zfstest kernel:  <TASK>
Oct 18 20:15:39 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:39 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:39 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:39 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:39 zfstest kernel:  ? zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:15:39 zfstest kernel:  ? zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:15:39 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:39 zfstest kernel:  zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:15:39 zfstest kernel:  zfs_acl_free+0xe/0x50 [zfs]
Oct 18 20:15:39 zfstest kernel:  zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:15:39 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:15:39 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:15:39 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:15:39 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:39 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:15:39 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:15:39 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:15:39 zfstest kernel:  do_shrink_slab+0x132/0x340
Oct 18 20:15:39 zfstest kernel:  shrink_slab+0x7b/0x3d0
Oct 18 20:15:39 zfstest kernel:  shrink_node+0x2e5/0xab0
Oct 18 20:15:39 zfstest kernel:  ? sched_clock_cpu+0x11/0x1a0
Oct 18 20:15:39 zfstest kernel:  balance_pgdat+0x376/0x8b0
Oct 18 20:15:39 zfstest kernel:  ? finish_task_switch+0x95/0x280
Oct 18 20:15:39 zfstest kernel:  ? __x64_sys_sched_getscheduler+0x80/0x80
Oct 18 20:15:39 zfstest kernel:  ? _raw_spin_lock_irqsave+0x24/0x50
Oct 18 20:15:39 zfstest kernel:  ? lock_timer_base+0x82/0xa0
Oct 18 20:15:39 zfstest kernel:  kswapd+0x1e0/0x350
Oct 18 20:15:39 zfstest kernel:  ? prepare_to_wait_event+0x110/0x110
Oct 18 20:15:39 zfstest kernel:  ? balance_pgdat+0x8b0/0x8b0
Oct 18 20:15:39 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:39 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:39 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:39 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:39 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:39 zfstest kernel:  </TASK>
Oct 18 20:15:39 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bec75fb6001-0xffff9bec75fb600f=0xcc
Oct 18 20:15:39 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bec75fb6040 not freed
Oct 18 20:15:39 zfstest kernel: =============================================================================
Oct 18 20:15:39 zfstest kernel: BUG kmalloc-32 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:39 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:39 zfstest kernel: 0xffff9bec2a1de000-0xffff9bec2a1de00f @offset=0. First byte 0xa5 instead of 0xcc
Oct 18 20:15:39 zfstest kernel: Slab 0xffffe87080a87780 objects=42 used=15 fp=0xffff9bec2a1de0e0 flags=0xfffffc0000200(workingset|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:39 zfstest kernel: Object 0xffff9bec2a1de020 @offset=32 fp=0xffff9bec2a1ded40
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec2a1de000: a5 8d 9c 69 d4 e0 83 10 b4 2a 86 cd f2 21 0d 58  ...i.....*...!.X
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec2a1de010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec2a1de020: 00 00 00 00 01 00 00 00 40 6c 34 0d ec 9b ff ff  ........@l4.....
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec2a1de030: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec2a1de040: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:39 zfstest kernel: Padding  ffff9bec2a1de050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:39 zfstest kernel: CPU: 59 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:39 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:39 zfstest kernel: Call Trace:
Oct 18 20:15:39 zfstest kernel:  <TASK>
Oct 18 20:15:39 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:39 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:39 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:39 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:39 zfstest kernel:  ? nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:39 zfstest kernel:  ? nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:39 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:39 zfstest kernel:  ? _raw_spin_unlock_irqrestore+0x23/0x40
Oct 18 20:15:39 zfstest kernel:  nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:39 zfstest kernel:  zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:15:39 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:15:39 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:15:39 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:15:39 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:39 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:15:39 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:15:39 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:15:39 zfstest kernel:  do_shrink_slab+0x132/0x340
Oct 18 20:15:39 zfstest kernel:  shrink_slab+0x7b/0x3d0
Oct 18 20:15:39 zfstest kernel:  shrink_node+0x2e5/0xab0
Oct 18 20:15:39 zfstest kernel:  ? sched_clock_cpu+0x11/0x1a0
Oct 18 20:15:39 zfstest kernel:  balance_pgdat+0x376/0x8b0
Oct 18 20:15:39 zfstest kernel:  ? finish_task_switch+0x95/0x280
Oct 18 20:15:39 zfstest kernel:  ? __x64_sys_sched_getscheduler+0x80/0x80
Oct 18 20:15:39 zfstest kernel:  ? _raw_spin_lock_irqsave+0x24/0x50
Oct 18 20:15:39 zfstest kernel:  ? lock_timer_base+0x82/0xa0
Oct 18 20:15:39 zfstest kernel:  kswapd+0x1e0/0x350
Oct 18 20:15:39 zfstest kernel:  ? prepare_to_wait_event+0x110/0x110
Oct 18 20:15:39 zfstest kernel:  ? balance_pgdat+0x8b0/0x8b0
Oct 18 20:15:39 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:39 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:39 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:39 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:39 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:39 zfstest kernel:  </TASK>
Oct 18 20:15:39 zfstest kernel: FIX kmalloc-32: Restoring Left Redzone 0xffff9bec2a1de000-0xffff9bec2a1de00f=0xcc
Oct 18 20:15:39 zfstest kernel: FIX kmalloc-32: Object at 0xffff9bec2a1de020 not freed
Oct 18 20:15:40 zfstest kernel: =============================================================================
Oct 18 20:15:40 zfstest kernel: BUG kmalloc-32 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:40 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:40 zfstest kernel: 0xffff9bec42641000-0xffff9bec4264100f @offset=0. First byte 0x19 instead of 0xcc
Oct 18 20:15:40 zfstest kernel: Slab 0xffffe87081099040 objects=42 used=19 fp=0xffff9bec42641bc0 flags=0xfffffc0000200(workingset|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:40 zfstest kernel: Object 0xffff9bec42641020 @offset=32 fp=0xffff9bec426415c0
Oct 18 20:15:40 zfstest kernel: Redzone  ffff9bec42641000: 19 ea 29 0d 99 e6 c5 03 4f e2 76 72 82 c8 c1 69  ..).....O.vr...i
Oct 18 20:15:40 zfstest kernel: Redzone  ffff9bec42641010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec42641020: 00 00 00 00 01 00 00 00 00 0d 8e 03 ec 9b ff ff  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec42641030: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:40 zfstest kernel: Redzone  ffff9bec42641040: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:40 zfstest kernel: Padding  ffff9bec42641050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:40 zfstest kernel: CPU: 100 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:40 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:40 zfstest kernel: Call Trace:
Oct 18 20:15:40 zfstest kernel:  <TASK>
Oct 18 20:15:40 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:40 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:40 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:40 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:40 zfstest kernel:  ? nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:40 zfstest kernel:  ? nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:40 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:40 zfstest kernel:  ? _raw_spin_unlock_irqrestore+0x23/0x40
Oct 18 20:15:40 zfstest kernel:  nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:40 zfstest kernel:  zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:15:40 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:15:40 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:15:40 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:15:40 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:40 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:15:40 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:15:40 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:15:40 zfstest kernel:  do_shrink_slab+0x132/0x340
Oct 18 20:15:40 zfstest kernel:  shrink_slab+0x7b/0x3d0
Oct 18 20:15:40 zfstest kernel:  shrink_node+0x2e5/0xab0
Oct 18 20:15:40 zfstest kernel:  ? sched_clock_cpu+0x11/0x1a0
Oct 18 20:15:40 zfstest kernel:  balance_pgdat+0x376/0x8b0
Oct 18 20:15:40 zfstest kernel:  ? finish_task_switch+0x95/0x280
Oct 18 20:15:40 zfstest kernel:  ? __x64_sys_sched_getscheduler+0x80/0x80
Oct 18 20:15:40 zfstest kernel:  ? _raw_spin_lock_irqsave+0x24/0x50
Oct 18 20:15:40 zfstest kernel:  ? lock_timer_base+0x82/0xa0
Oct 18 20:15:40 zfstest kernel:  kswapd+0x1e0/0x350
Oct 18 20:15:40 zfstest kernel:  ? prepare_to_wait_event+0x110/0x110
Oct 18 20:15:40 zfstest kernel:  ? balance_pgdat+0x8b0/0x8b0
Oct 18 20:15:40 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:40 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:40 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:40 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:40 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:40 zfstest kernel:  </TASK>
Oct 18 20:15:40 zfstest kernel: FIX kmalloc-32: Restoring Left Redzone 0xffff9bec42641000-0xffff9bec4264100f=0xcc
Oct 18 20:15:40 zfstest kernel: FIX kmalloc-32: Object at 0xffff9bec42641020 not freed
Oct 18 20:15:40 zfstest kernel: =============================================================================
Oct 18 20:15:40 zfstest kernel: BUG dnode_t (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:40 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:40 zfstest kernel: 0xffff9bec70c88000-0xffff9bec70c88007 @offset=0. First byte 0xe7 instead of 0xcc
Oct 18 20:15:40 zfstest kernel: Slab 0xffffe87081c32200 objects=32 used=8 fp=0xffff9bec70c8df48 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:40 zfstest kernel: Object 0xffff9bec70c88008 @offset=8 fp=0xffff9bec70c887f8
Oct 18 20:15:40 zfstest kernel: Redzone  ffff9bec70c88000: e7 5e 95 6c 82 98 38 33                          .^.l..83
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88008: 92 8c b1 6f 74 7c 68 54 c1 33 2d 21 ed 9b ff ff  ...ot|hT.3-!....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88018: 00 00 00 00 00 00 00 00 20 80 c8 70 ec 9b ff ff  ........ ..p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88028: 20 80 c8 70 ec 9b ff ff 00 00 00 00 00 00 00 00   ..p............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88038: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88048: 01 d0 b8 33 ed 9b ff ff 63 9e 00 00 00 00 00 00  ...3....c.......
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88058: 78 a4 ee eb ed 9b ff ff 38 e2 21 4a ec 9b ff ff  x.......8.!J....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88068: 00 06 b9 60 ed 9b ff ff 13 00 00 00 b0 00 2c 01  ...`..........,.
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88078: 00 00 01 11 0a 00 02 00 00 04 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88088: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88098: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880a8: 00 6b 00 00 00 00 00 00 00 00 6b 6b 00 00 00 00  .k........kk....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880b8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880e8: 00 00 00 00 6b 6b 6b 6b 00 01 00 00 00 00 ad de  ....kkkk........
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880f8: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88108: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88118: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88128: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88138: 00 00 00 00 00 00 00 00 40 81 c8 70 ec 9b ff ff  [email protected]....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88148: 40 81 c8 70 ec 9b ff ff 00 00 00 00 6b 6b 6b 6b  @..p........kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88158: 00 00 00 00 00 00 00 00 98 01 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88168: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88178: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88188: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88198: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881a8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881b8: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881c8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881d8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88208: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88218: 00 00 00 00 00 00 00 00 f5 45 65 34 6b 6b 6b 6b  .........Ee4kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88228: 00 00 00 00 6b 6b 6b 6b 30 82 c8 70 ec 9b ff ff  ....kkkk0..p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88238: 30 82 c8 70 ec 9b ff ff 00 00 00 00 6b 6b 6b 6b  0..p........kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88248: 48 82 c8 70 ec 9b ff ff 48 82 c8 70 ec 9b ff ff  H..p....H..p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88258: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88268: f5 45 65 34 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b  .Ee4kkkk....kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88278: 78 82 c8 70 ec 9b ff ff 78 82 c8 70 ec 9b ff ff  x..p....x..p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88288: 00 00 00 00 6b 6b 6b 6b 90 82 c8 70 ec 9b ff ff  ....kkkk...p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88298: 90 82 c8 70 ec 9b ff ff 00 00 00 00 00 00 00 00  ...p............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882a8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882b8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882d8: 00 00 00 00 00 00 00 00 e0 82 c8 70 ec 9b ff ff  ...........p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882e8: e0 82 c8 70 ec 9b ff ff 00 00 00 00 6b 6b 6b 6b  ...p........kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88308: 20 85 54 c0 ff ff ff ff 40 00 00 00 00 00 00 00   .T.....@.......
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88318: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88328: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88338: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88348: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88358: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88368: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88378: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88388: 00 00 00 00 00 00 00 00 90 83 c8 70 ec 9b ff ff  ...........p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88398: 90 83 c8 70 ec 9b ff ff 00 00 00 00 6b 6b 6b 6b  ...p........kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c883a8: 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c883b8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c883c8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c883d8: 00 00 00 00 6b 6b 6b a5                          ....kkk.
Oct 18 20:15:40 zfstest kernel: Redzone  ffff9bec70c883e0: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:40 zfstest kernel: Padding  ffff9bec70c883f0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:40 zfstest kernel: CPU: 115 PID: 1498 Comm: dbu_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:40 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:40 zfstest kernel: Call Trace:
Oct 18 20:15:40 zfstest kernel:  <TASK>
Oct 18 20:15:40 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:40 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:40 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:40 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:40 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:40 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:40 zfstest kernel:  kmem_cache_free+0x354/0x3f0
Oct 18 20:15:40 zfstest kernel:  spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:40 zfstest kernel:  dnode_destroy+0x1df/0x240 [zfs]
Oct 18 20:15:40 zfstest kernel:  dnode_buf_evict_async+0x75/0xd0 [zfs]
Oct 18 20:15:40 zfstest kernel:  taskq_thread+0x259/0x450 [spl]
Oct 18 20:15:40 zfstest kernel:  ? wake_up_state+0x10/0x10
Oct 18 20:15:40 zfstest kernel:  ? dnode_create+0x1e0/0x1e0 [zfs]
Oct 18 20:15:40 zfstest kernel:  ? task_done+0x90/0x90 [spl]
Oct 18 20:15:40 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:40 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:40 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:40 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:40 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:40 zfstest kernel:  </TASK>
Oct 18 20:15:40 zfstest kernel: FIX dnode_t: Restoring Left Redzone 0xffff9bec70c88000-0xffff9bec70c88007=0xcc
Oct 18 20:15:40 zfstest kernel: FIX dnode_t: Object at 0xffff9bec70c88008 not freed
Oct 18 20:15:47 zfstest kernel: =============================================================================
Oct 18 20:15:47 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:47 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:47 zfstest kernel: 0xffff9bedac286000-0xffff9bedac28600f @offset=0. First byte 0x10 instead of 0xcc
Oct 18 20:15:47 zfstest kernel: Slab 0xffffe87086b0a180 objects=42 used=34 fp=0xffff9bedac286580 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:15:47 zfstest kernel: Object 0xffff9bedac286040 @offset=64 fp=0x0000000000000000
Oct 18 20:15:47 zfstest kernel: Redzone  ffff9bedac286000: 10 4a 12 08 0a 98 5b 30 88 63 7a 79 75 11 48 15  .J....[0.czyu.H.
Oct 18 20:15:47 zfstest kernel: Redzone  ffff9bedac286010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:47 zfstest kernel: Redzone  ffff9bedac286020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:47 zfstest kernel: Redzone  ffff9bedac286030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:47 zfstest kernel: Object   ffff9bedac286040: 00 00 00 10 9f 01 1e 00 00 00 40 20 89 00 12 00  ..........@ ....
Oct 18 20:15:47 zfstest kernel: Object   ffff9bedac286050: 00 00 00 40 89 00 12 00 6b 6b 6b 6b 6b 6b 6b 6b  [email protected]
Oct 18 20:15:47 zfstest kernel: Object   ffff9bedac286060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:47 zfstest kernel: Object   ffff9bedac286070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:15:47 zfstest kernel: Redzone  ffff9bedac286080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:47 zfstest kernel: Padding  ffff9bedac286090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:47 zfstest kernel: Padding  ffff9bedac2860a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:47 zfstest kernel: Padding  ffff9bedac2860b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:47 zfstest kernel: CPU: 114 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:47 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:47 zfstest kernel: Call Trace:
Oct 18 20:15:47 zfstest kernel:  <TASK>
Oct 18 20:15:47 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:47 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:47 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:47 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:47 zfstest kernel:  ? zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:15:47 zfstest kernel:  ? zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:15:47 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:47 zfstest kernel:  zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:15:47 zfstest kernel:  zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:15:47 zfstest kernel:  zfs_acl_free+0xe/0x50 [zfs]
Oct 18 20:15:47 zfstest kernel:  zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:15:47 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:15:47 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:15:47 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:15:47 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:47 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:15:47 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:15:47 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:15:47 zfstest kernel:  do_shrink_slab+0x132/0x340
Oct 18 20:15:47 zfstest kernel:  shrink_slab+0x7b/0x3d0
Oct 18 20:15:47 zfstest kernel:  shrink_node+0x2e5/0xab0
Oct 18 20:15:47 zfstest kernel:  balance_pgdat+0x376/0x8b0
Oct 18 20:15:47 zfstest kernel:  ? finish_task_switch+0x95/0x280
Oct 18 20:15:47 zfstest kernel:  ? __x64_sys_sched_getscheduler+0x80/0x80
Oct 18 20:15:47 zfstest kernel:  ? _raw_spin_lock_irqsave+0x24/0x50
Oct 18 20:15:47 zfstest kernel:  ? lock_timer_base+0x82/0xa0
Oct 18 20:15:47 zfstest kernel:  kswapd+0x1e0/0x350
Oct 18 20:15:47 zfstest kernel:  ? prepare_to_wait_event+0x110/0x110
Oct 18 20:15:47 zfstest kernel:  ? balance_pgdat+0x8b0/0x8b0
Oct 18 20:15:47 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:47 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:47 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:47 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:47 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:47 zfstest kernel:  </TASK>
Oct 18 20:15:47 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bedac286000-0xffff9bedac28600f=0xcc
Oct 18 20:15:47 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bedac286040 not freed
Oct 18 20:15:47 zfstest systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
Oct 18 20:15:48 zfstest kernel: =============================================================================
Oct 18 20:15:48 zfstest kernel: BUG lsm_inode_cache (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:48 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:48 zfstest kernel: 0xffff9bedfa59c000-0xffff9bedfa59c007 @offset=0. First byte 0x10 instead of 0xcc
Oct 18 20:15:48 zfstest kernel: Slab 0xffffe87087e96700 objects=42 used=35 fp=0xffff9bedfa59c548 flags=0x17ffffc0000200(workingset|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:15:48 zfstest kernel: Object 0xffff9bedfa59c008 @offset=8 fp=0xffff9bedfa59c8a8
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bedfa59c000: 10 d0 76 2a 55 2b 96 f8                          ..v*U+..
Oct 18 20:15:48 zfstest kernel: Object   ffff9bedfa59c008: 88 c1 27 43 ee 9b ff ff 00 00 00 00 00 00 00 00  ..'C............
Oct 18 20:15:48 zfstest kernel: Object   ffff9bedfa59c018: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:48 zfstest kernel: Object   ffff9bedfa59c028: 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:48 zfstest kernel: Object   ffff9bedfa59c038: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bedfa59c048: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:48 zfstest kernel: Padding  ffff9bedfa59c058: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:48 zfstest kernel: CPU: 47 PID: 304 Comm: ksoftirqd/47 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:48 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:48 zfstest kernel: Call Trace:
Oct 18 20:15:48 zfstest kernel:  <TASK>
Oct 18 20:15:48 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:48 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:48 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:48 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:48 zfstest kernel:  ? rcu_core+0x430/0x770
Oct 18 20:15:48 zfstest kernel:  ? rcu_core+0x430/0x770
Oct 18 20:15:48 zfstest kernel:  kmem_cache_free+0x354/0x3f0
Oct 18 20:15:48 zfstest kernel:  ? rcu_core+0x3d0/0x770
Oct 18 20:15:48 zfstest kernel:  rcu_core+0x430/0x770
Oct 18 20:15:48 zfstest kernel:  ? try_to_wake_up+0x98/0x5f0
Oct 18 20:15:48 zfstest kernel:  handle_softirqs+0xd6/0x2a0
Oct 18 20:15:48 zfstest kernel:  run_ksoftirqd+0x33/0x40
Oct 18 20:15:48 zfstest kernel:  smpboot_thread_fn+0x175/0x210
Oct 18 20:15:48 zfstest kernel:  ? sort_range+0x20/0x20
Oct 18 20:15:48 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:48 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:48 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:48 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:48 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:48 zfstest kernel:  </TASK>
Oct 18 20:15:48 zfstest kernel: FIX lsm_inode_cache: Restoring Left Redzone 0xffff9bedfa59c000-0xffff9bedfa59c007=0xcc
Oct 18 20:15:48 zfstest kernel: FIX lsm_inode_cache: Object at 0xffff9bedfa59c008 not freed
Oct 18 20:15:48 zfstest kernel: =============================================================================
Oct 18 20:15:48 zfstest kernel: BUG kmalloc-32 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:48 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:48 zfstest kernel: 0xffff9bec6a515000-0xffff9bec6a51500f @offset=0. First byte 0x43 instead of 0xcc
Oct 18 20:15:48 zfstest kernel: Slab 0xffffe87081a94540 objects=42 used=5 fp=0xffff9bec6a515860 flags=0xfffffc0000200(workingset|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:48 zfstest kernel: Object 0xffff9bec6a515020 @offset=32 fp=0xffff9bec6a515260
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec6a515000: 43 11 1c b1 f1 1f 59 f5 59 e8 47 1c 93 06 95 df  C.....Y.Y.G.....
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec6a515010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec6a515020: 00 00 00 00 01 00 00 00 80 fa 64 30 ec 9b ff ff  ..........d0....
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec6a515030: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec6a515040: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:48 zfstest kernel: Padding  ffff9bec6a515050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:48 zfstest kernel: CPU: 34 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:48 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:48 zfstest kernel: Call Trace:
Oct 18 20:15:48 zfstest kernel:  <TASK>
Oct 18 20:15:48 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:48 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:48 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:48 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:48 zfstest kernel:  ? nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:48 zfstest kernel:  ? nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:48 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:48 zfstest kernel:  ? _raw_spin_unlock_irqrestore+0x23/0x40
Oct 18 20:15:48 zfstest kernel:  nvlist_free+0xce/0xe0 [zfs]
Oct 18 20:15:48 zfstest kernel:  zfs_inode_destroy+0xc5/0xf0 [zfs]
Oct 18 20:15:48 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:15:48 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:15:48 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:15:48 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:48 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:15:48 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:15:48 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:15:48 zfstest kernel:  do_shrink_slab+0x132/0x340
Oct 18 20:15:48 zfstest kernel:  shrink_slab+0x7b/0x3d0
Oct 18 20:15:48 zfstest kernel:  shrink_node+0x2e5/0xab0
Oct 18 20:15:48 zfstest kernel:  ? sched_clock_cpu+0x11/0x1a0
Oct 18 20:15:48 zfstest kernel:  balance_pgdat+0x376/0x8b0
Oct 18 20:15:48 zfstest kernel:  ? finish_task_switch+0x95/0x280
Oct 18 20:15:48 zfstest kernel:  ? __x64_sys_sched_getscheduler+0x80/0x80
Oct 18 20:15:48 zfstest kernel:  ? _raw_spin_lock_irqsave+0x24/0x50
Oct 18 20:15:48 zfstest kernel:  ? lock_timer_base+0x82/0xa0
Oct 18 20:15:48 zfstest kernel:  kswapd+0x1e0/0x350
Oct 18 20:15:48 zfstest kernel:  ? prepare_to_wait_event+0x110/0x110
Oct 18 20:15:48 zfstest kernel:  ? balance_pgdat+0x8b0/0x8b0
Oct 18 20:15:48 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:48 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:48 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:48 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:48 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:48 zfstest kernel:  </TASK>
Oct 18 20:15:48 zfstest kernel: FIX kmalloc-32: Restoring Left Redzone 0xffff9bec6a515000-0xffff9bec6a51500f=0xcc
Oct 18 20:15:48 zfstest kernel: FIX kmalloc-32: Object at 0xffff9bec6a515020 not freed
Oct 18 20:15:48 zfstest systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
Oct 18 20:15:48 zfstest systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
Oct 18 20:15:48 zfstest kernel: =============================================================================
Oct 18 20:15:48 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:48 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:48 zfstest kernel: 0xffff9bec4a922000-0xffff9bec4a92200f @offset=0. First byte 0xa instead of 0xcc
Oct 18 20:15:48 zfstest kernel: Slab 0xffffe870812a4880 objects=42 used=22 fp=0xffff9bec4a922d00 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:48 zfstest kernel: Object 0xffff9bec4a922040 @offset=64 fp=0xffff9bec4a922580
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec4a922000: 0a 98 12 9a be 27 4a db 3f e5 87 62 61 71 3a 86  .....'J.?..baq:.
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec4a922010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec4a922020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec4a922030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec4a922040: 00 00 00 10 9f 01 1e 00 00 00 40 20 89 00 12 00  ..........@ ....
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec4a922050: 00 00 00 40 89 00 12 00 6b 6b 6b 6b 6b 6b 6b 6b  [email protected]
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec4a922060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec4a922070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec4a922080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:48 zfstest kernel: Padding  ffff9bec4a922090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:48 zfstest kernel: Padding  ffff9bec4a9220a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:48 zfstest kernel: Padding  ffff9bec4a9220b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:48 zfstest kernel: CPU: 34 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:48 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:48 zfstest kernel: Call Trace:
Oct 18 20:15:48 zfstest kernel:  <TASK>
Oct 18 20:15:48 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:48 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:48 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:48 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:48 zfstest kernel:  ? zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:15:48 zfstest kernel:  ? zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:15:48 zfstest kernel:  kfree+0x260/0x2f0
Oct 18 20:15:48 zfstest kernel:  zfs_acl_node_free+0x29/0x40 [zfs]
Oct 18 20:15:48 zfstest kernel:  zfs_acl_release_nodes+0x4f/0x70 [zfs]
Oct 18 20:15:48 zfstest kernel:  zfs_acl_free+0xe/0x50 [zfs]
Oct 18 20:15:48 zfstest kernel:  zfs_inode_destroy+0xaf/0xf0 [zfs]
Oct 18 20:15:48 zfstest kernel:  destroy_inode+0x32/0x70
Oct 18 20:15:48 zfstest kernel:  evict+0x15e/0x260
Oct 18 20:15:48 zfstest kernel:  ? unlock_new_inode+0x60/0x60
Oct 18 20:15:48 zfstest kernel:  ? _raw_spin_unlock+0x15/0x30
Oct 18 20:15:48 zfstest kernel:  dispose_list+0x32/0x50
Oct 18 20:15:48 zfstest kernel:  prune_icache_sb+0x54/0x80
Oct 18 20:15:48 zfstest kernel:  super_cache_scan+0x110/0x180
Oct 18 20:15:48 zfstest kernel:  do_shrink_slab+0x132/0x340
Oct 18 20:15:48 zfstest kernel:  shrink_slab+0x7b/0x3d0
Oct 18 20:15:48 zfstest kernel:  shrink_node+0x2e5/0xab0
Oct 18 20:15:48 zfstest kernel:  ? sched_clock_cpu+0x11/0x1a0
Oct 18 20:15:48 zfstest kernel:  balance_pgdat+0x376/0x8b0
Oct 18 20:15:48 zfstest kernel:  ? finish_task_switch+0x95/0x280
Oct 18 20:15:48 zfstest kernel:  ? __x64_sys_sched_getscheduler+0x80/0x80
Oct 18 20:15:48 zfstest kernel:  ? _raw_spin_lock_irqsave+0x24/0x50
Oct 18 20:15:48 zfstest kernel:  ? lock_timer_base+0x82/0xa0
Oct 18 20:15:48 zfstest kernel:  kswapd+0x1e0/0x350
Oct 18 20:15:48 zfstest kernel:  ? prepare_to_wait_event+0x110/0x110
Oct 18 20:15:48 zfstest kernel:  ? balance_pgdat+0x8b0/0x8b0
Oct 18 20:15:48 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:48 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:48 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:48 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:48 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:48 zfstest kernel:  </TASK>
Oct 18 20:15:48 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bec4a922000-0xffff9bec4a92200f=0xcc
Oct 18 20:15:48 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bec4a922040 not freed
Oct 18 20:15:50 zfstest kernel: =============================================================================
Oct 18 20:15:50 zfstest kernel: BUG zio_buf_comb_512 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:50 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:50 zfstest kernel: 0xffff9bee08818000-0xffff9bee0881800f @offset=0. First byte 0x9 instead of 0xcc
Oct 18 20:15:50 zfstest kernel: Slab 0xffffe87088220600 objects=21 used=6 fp=0xffff9bee08818800 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:15:50 zfstest kernel: Object 0xffff9bee08818200 @offset=512 fp=0xffff9bee0881ca00
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818000: 09 fd 72 47 92 66 0d 2f 45 e4 ca b9 cc 5f 0a 50  ..rG.f./E...._.P
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818040: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818050: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818060: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818070: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818080: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818090: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818100: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818110: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818120: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818130: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818140: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818150: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818160: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818170: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818180: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818190: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818200: 69 6d 70 6f 72 74 20 5f 70 6c 6f 74 6c 79 5f 75  import _plotly_u
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818210: 74 69 6c 73 2e 62 61 73 65 76 61 6c 69 64 61 74  tils.basevalidat
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818220: 6f 72 73 0a 0a 0a 63 6c 61 73 73 20 53 6f 75 72  ors...class Sour
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818230: 63 65 56 61 6c 69 64 61 74 6f 72 28 5f 70 6c 6f  ceValidator(_plo
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818240: 74 6c 79 5f 75 74 69 6c 73 2e 62 61 73 65 76 61  tly_utils.baseva
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818250: 6c 69 64 61 74 6f 72 73 2e 41 6e 79 56 61 6c 69  lidators.AnyVali
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818260: 64 61 74 6f 72 29 3a 0a 20 20 20 20 64 65 66 20  dator):.    def
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818270: 5f 5f 69 6e 69 74 5f 5f 28 73 65 6c 66 2c 20 70  __init__(self, p
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818280: 6c 6f 74 6c 79 5f 6e 61 6d 65 3d 22 73 6f 75 72  lotly_name="sour
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818290: 63 65 22 2c 20 70 61 72 65 6e 74 5f 6e 61 6d 65  ce", parent_name
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182a0: 3d 22 6c 61 79 6f 75 74 2e 6d 61 70 2e 6c 61 79  ="layout.map.lay
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182b0: 65 72 22 2c 20 2a 2a 6b 77 61 72 67 73 29 3a 0a  er", **kwargs):.
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182c0: 20 20 20 20 20 20 20 20 73 75 70 65 72 28 53 6f          super(So
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182d0: 75 72 63 65 56 61 6c 69 64 61 74 6f 72 2c 20 73  urceValidator, s
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182e0: 65 6c 66 29 2e 5f 5f 69 6e 69 74 5f 5f 28 0a 20  elf).__init__(.
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182f0: 20 20 20 20 20 20 20 20 20 20 20 70 6c 6f 74 6c             plotl
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818300: 79 5f 6e 61 6d 65 3d 70 6c 6f 74 6c 79 5f 6e 61  y_name=plotly_na
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818310: 6d 65 2c 0a 20 20 20 20 20 20 20 20 20 20 20 20  me,.
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818320: 70 61 72 65 6e 74 5f 6e 61 6d 65 3d 70 61 72 65  parent_name=pare
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818330: 6e 74 5f 6e 61 6d 65 2c 0a 20 20 20 20 20 20 20  nt_name,.
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818340: 20 20 20 20 20 65 64 69 74 5f 74 79 70 65 3d 6b       edit_type=k
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818350: 77 61 72 67 73 2e 70 6f 70 28 22 65 64 69 74 5f  wargs.pop("edit_
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818360: 74 79 70 65 22 2c 20 22 70 6c 6f 74 22 29 2c 0a  type", "plot"),.
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818370: 20 20 20 20 20 20 20 20 20 20 20 20 2a 2a 6b 77              **kw
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818380: 61 72 67 73 2c 0a 20 20 20 20 20 20 20 20 29 0a  args,.        ).
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818400: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818410: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818420: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818430: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818440: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818450: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818460: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818470: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818480: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818490: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818500: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818510: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818520: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818530: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818540: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818550: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818560: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818570: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818580: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818590: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: CPU: 66 PID: 1496 Comm: arc_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:50 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:50 zfstest kernel: Call Trace:
Oct 18 20:15:50 zfstest kernel:  <TASK>
Oct 18 20:15:50 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:50 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:50 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:50 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:50 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:50 zfstest kernel:  ? spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:50 zfstest kernel:  kmem_cache_free+0x354/0x3f0
Oct 18 20:15:50 zfstest kernel:  spl_kmem_cache_free+0x12b/0x1e0 [spl]
Oct 18 20:15:50 zfstest kernel:  abd_free+0x15a/0x190 [zfs]
Oct 18 20:15:50 zfstest kernel:  ? aggsum_add+0x17a/0x190 [zfs]
Oct 18 20:15:50 zfstest kernel:  arc_hdr_free_abd+0xfe/0x280 [zfs]
Oct 18 20:15:50 zfstest kernel:  arc_evict_state+0x35c/0x790 [zfs]
Oct 18 20:15:50 zfstest kernel:  arc_evict_cb+0x4e7/0x750 [zfs]
Oct 18 20:15:50 zfstest kernel:  zthr_procedure+0x12e/0x140 [zfs]
Oct 18 20:15:50 zfstest kernel:  ? zrl_is_locked+0x20/0x20 [zfs]
Oct 18 20:15:50 zfstest kernel:  ? spl_taskq_fini+0x70/0x70 [spl]
Oct 18 20:15:50 zfstest kernel:  thread_generic_wrapper+0x55/0x60 [spl]
Oct 18 20:15:50 zfstest kernel:  kthread+0xdb/0x110
Oct 18 20:15:50 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:50 zfstest kernel:  ret_from_fork+0x2d/0x50
Oct 18 20:15:50 zfstest kernel:  ? kthread_park+0x80/0x80
Oct 18 20:15:50 zfstest kernel:  ret_from_fork_asm+0x11/0x20
Oct 18 20:15:50 zfstest kernel:  </TASK>
Oct 18 20:15:50 zfstest kernel: FIX zio_buf_comb_512: Restoring Left Redzone 0xffff9bee08818000-0xffff9bee0881800f=0xcc
Oct 18 20:15:50 zfstest kernel: FIX zio_buf_comb_512: Object at 0xffff9bee08818200 not freed
Oct 18 20:15:54 zfstest kernel: =============================================================================
Oct 18 20:15:54 zfstest kernel: BUG lsm_inode_cache (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:54 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:54 zfstest kernel: 0xffff9bec0398f000-0xffff9bec0398f007 @offset=0. First byte 0x84 instead of 0xcc
Oct 18 20:15:54 zfstest kernel: Slab 0xffffe870800e63c0 objects=42 used=42 fp=0x0000000000000000 flags=0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:54 zfstest kernel: Object 0xffff9bec0398f008 @offset=8 fp=0xffff9bec0398f428
Oct 18 20:15:54 zfstest kernel: Redzone  ffff9bec0398f000: 84 a0 bb 2b 7d 43 67 7e                          ...+}Cg~
Oct 18 20:15:54 zfstest kernel: Object   ffff9bec0398f008: 08 7f b2 33 ec 9b ff ff 00 00 00 00 00 00 00 00  ...3............
Oct 18 20:15:54 zfstest kernel: Object   ffff9bec0398f018: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:54 zfstest kernel: Object   ffff9bec0398f028: 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:54 zfstest kernel: Object   ffff9bec0398f038: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:54 zfstest kernel: Redzone  ffff9bec0398f048: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:54 zfstest kernel: Padding  ffff9bec0398f058: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:54 zfstest kernel: CPU: 15 PID: 0 Comm: swapper/15 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:54 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:54 zfstest kernel: Call Trace:
Oct 18 20:15:54 zfstest kernel:  <IRQ>
Oct 18 20:15:54 zfstest kernel:  dump_stack_lvl+0x62/0x80
Oct 18 20:15:54 zfstest kernel:  check_bytes_and_report+0x106/0x130
Oct 18 20:15:54 zfstest kernel:  check_object+0x1d9/0x360
Oct 18 20:15:54 zfstest kernel:  free_to_partial_list+0x1cf/0x520
Oct 18 20:15:54 zfstest kernel:  ? rcu_core+0x430/0x770
Oct 18 20:15:54 zfstest kernel:  ? rcu_core+0x430/0x770
Oct 18 20:15:54 zfstest kernel:  kmem_cache_free+0x354/0x3f0
Oct 18 20:15:54 zfstest kernel:  ? rcu_core+0x3d0/0x770
Oct 18 20:15:54 zfstest kernel:  rcu_core+0x430/0x770
Oct 18 20:15:54 zfstest kernel:  ? kvm_clock_read+0x14/0x30
Oct 18 20:15:54 zfstest kernel:  handle_softirqs+0xd6/0x2a0
Oct 18 20:15:54 zfstest kernel:  irq_exit_rcu+0x6f/0xa0
Oct 18 20:15:54 zfstest kernel:  sysvec_apic_timer_interrupt+0xa1/0xd0
Oct 18 20:15:54 zfstest kernel:  </IRQ>
Oct 18 20:15:54 zfstest kernel:  <TASK>
Oct 18 20:15:54 zfstest kernel:  asm_sysvec_apic_timer_interrupt+0x16/0x20
Oct 18 20:15:54 zfstest kernel: RIP: 0010:pv_native_safe_halt+0xb/0x10
Oct 18 20:15:54 zfstest kernel: Code: 09 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 22 d7 c3 66 66 2e 0f 1f 84 00 00 00 00 00 90 eb 07 0f 00 2d 0f fd 2c 00 fb f4 <c3> 0f 1f 40 00 83 ff 07 77 21 89 ff ff 24 fd f8 2e e1 b8 0f 21 f8
Oct 18 20:15:54 zfstest kernel: RSP: 0018:ffffbea8c0137ec0 EFLAGS: 00000202
Oct 18 20:15:54 zfstest kernel: RAX: 000000000000000f RBX: 000000000000000f RCX: 0000000000000000
Oct 18 20:15:54 zfstest kernel: RDX: 0000000000000000 RSI: ffffffffb9247b2c RDI: 0000000000371d74
Oct 18 20:15:54 zfstest kernel: RBP: ffff9bed00becd80 R08: 0000001881573345 R09: 0000000000000000
Oct 18 20:15:54 zfstest kernel: R10: 00000000000001c1 R11: 0000000000000007 R12: 0000000000000000
Oct 18 20:15:54 zfstest kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Oct 18 20:15:54 zfstest kernel:  default_idle+0x5/0x10
Oct 18 20:15:54 zfstest kernel:  default_idle_call+0x2a/0xf0
Oct 18 20:15:54 zfstest kernel:  do_idle+0x1ea/0x230
Oct 18 20:15:54 zfstest kernel:  cpu_startup_entry+0x25/0x30
Oct 18 20:15:54 zfstest kernel:  start_secondary+0x101/0x120
Oct 18 20:15:54 zfstest kernel:  common_startup_64+0x13e/0x141
Oct 18 20:15:54 zfstest kernel:  </TASK>
Oct 18 20:15:54 zfstest kernel: FIX lsm_inode_cache: Restoring Left Redzone 0xffff9bec0398f000-0xffff9bec0398f007=0xcc
Oct 18 20:15:54 zfstest kernel: FIX lsm_inode_cache: Object at 0xffff9bec0398f008 not freed

Oct 19 '24 00:10 TheUbuntuGuy

...and here are the traces after running decode_stacktrace.sh after rebuilding ZFS (2.2.6) with debug symbols. I seem to be able to reliably reproduce on my system, so let me know if there is anything additional I can do to help.

Trace 1 decoded

Oct 18 20:08:37 zfstest kernel: =============================================================================
Oct 18 20:08:37 zfstest kernel: BUG dnode_t (Tainted: P           O      ): Left Redzone overwritten
Oct 18 20:08:37 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:37 zfstest kernel: 0xffff8d800aae8000-0xffff8d800aae8007 @offset=0. First byte 0xec instead of 0xcc
Oct 18 20:08:37 zfstest kernel: Slab 0xffffd668882aba00 objects=32 used=30 fp=0xffff8d800aaeabb0 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:37 zfstest kernel: Object 0xffff8d800aae8008 @offset=8 fp=0xffff8d800aaed360
Oct 18 20:08:37 zfstest kernel: Redzone  ffff8d800aae8000: ec f3 9e 56 c7 b0 24 32                          ...V..$2
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8008: 89 e4 3e e8 98 16 f5 be c1 33 51 15 7f 8d ff ff  ..>......3Q.....
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8018: 00 00 00 00 00 00 00 00 20 80 ae 0a 80 8d ff ff  ........ .......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8028: 20 80 ae 0a 80 8d ff ff 00 00 00 00 00 00 00 00   ...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8038: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8048: 01 50 5c 96 7f 8d ff ff b4 db 00 00 00 00 00 00  .P.............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8058: 68 ae f5 ad 7f 8d ff ff 40 6b f9 e0 7f 8d ff ff  h.......@k......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8068: 00 e8 fd 6e 80 8d ff ff 13 00 00 00 b0 00 2c 01  ...n..........,.
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8078: 00 00 01 11 00 00 0a 00 00 14 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8088: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8098: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80a8: 00 6b 00 00 00 00 00 00 00 00 6b 6b 00 00 00 00  .k........kk....
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80b8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80e8: 00 00 00 00 6b 6b 6b 6b 00 01 00 00 00 00 ad de  ....kkkk........
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae80f8: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8108: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8118: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8128: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8138: 00 00 00 00 00 00 00 00 40 81 ae 0a 80 8d ff ff  ........@.......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8148: 40 81 ae 0a 80 8d ff ff 00 00 00 00 6b 6b 6b 6b  @...........kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8158: 00 00 00 00 00 00 00 00 98 01 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8168: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8178: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8188: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8198: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81a8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81b8: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81c8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81d8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae81f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8208: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8218: 00 00 00 00 00 00 00 00 f5 45 65 34 6b 6b 6b 6b  .........Ee4kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8228: 00 00 00 00 6b 6b 6b 6b 30 82 ae 0a 80 8d ff ff  ....kkkk0.......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8238: 30 82 ae 0a 80 8d ff ff 00 00 00 00 6b 6b 6b 6b  0...........kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8248: 48 82 ae 0a 80 8d ff ff 48 82 ae 0a 80 8d ff ff  H.......H.......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8258: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8268: f5 45 65 34 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b  .Ee4kkkk....kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8278: 78 82 ae 0a 80 8d ff ff 78 82 ae 0a 80 8d ff ff  x.......x.......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8288: 00 00 00 00 6b 6b 6b 6b 90 82 ae 0a 80 8d ff ff  ....kkkk........
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8298: 90 82 ae 0a 80 8d ff ff 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82a8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82b8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82d8: 00 00 00 00 00 00 00 00 e0 82 ae 0a 80 8d ff ff  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82e8: e0 82 ae 0a 80 8d ff ff 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae82f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8308: 20 b5 51 c0 ff ff ff ff 40 00 00 00 00 00 00 00   .Q.....@.......
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8318: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8328: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8338: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8348: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8358: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8368: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8378: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8388: 00 00 00 00 00 00 00 00 90 83 ae 0a 80 8d ff ff  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae8398: 90 83 ae 0a 80 8d ff ff 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae83a8: 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae83b8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae83c8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:37 zfstest kernel: Object   ffff8d800aae83d8: 00 00 00 00 6b 6b 6b a5                          ....kkk.
Oct 18 20:08:37 zfstest kernel: Redzone  ffff8d800aae83e0: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:37 zfstest kernel: Padding  ffff8d800aae83f0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:08:37 zfstest kernel: CPU: 92 PID: 1503 Comm: dbu_evict Tainted: P           O       6.10.14-xxx #1
Oct 18 20:08:37 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:37 zfstest kernel: Call Trace:
Oct 18 20:08:37 zfstest kernel:  <TASK>
Oct 18 20:08:37 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:08:37 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:08:37 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:08:37 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:08:37 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:08:37 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:08:37 zfstest kernel: kmem_cache_free (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4523)
Oct 18 20:08:37 zfstest kernel: spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:08:37 zfstest kernel: dnode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:648 (discriminator 1)) zfs
Oct 18 20:08:37 zfstest kernel: dnode_buf_evict_async (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1351) zfs
Oct 18 20:08:37 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:08:37 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:08:37 zfstest kernel: ? dnode_create (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1947) zfs
Oct 18 20:08:37 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:08:37 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:08:37 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:37 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:08:37 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:37 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:08:37 zfstest kernel:  </TASK>
Oct 18 20:08:37 zfstest kernel: FIX dnode_t: Restoring Left Redzone 0xffff8d800aae8000-0xffff8d800aae8007=0xcc
Oct 18 20:08:37 zfstest kernel: FIX dnode_t: Object at 0xffff8d800aae8008 not freed
Oct 18 20:08:38 zfstest kernel: =============================================================================
Oct 18 20:08:38 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:38 zfstest kernel: 0xffff8d7f417ae000-0xffff8d7f417ae00f @offset=0. First byte 0xb4 instead of 0xcc
Oct 18 20:08:38 zfstest kernel: Slab 0xffffd6688505eb80 objects=42 used=42 fp=0x0000000000000000 flags=0x17ffffc0000040(head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:38 zfstest kernel: Object 0xffff8d7f417ae040 @offset=64 fp=0x0000000000000000
Oct 18 20:08:38 zfstest kernel: Redzone  ffff8d7f417ae000: b4 d6 64 fb 00 bd 27 7c 1b f6 9b 8f 2a c3 eb 15  ..d...'|....*...
Oct 18 20:08:38 zfstest kernel: Redzone  ffff8d7f417ae010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:38 zfstest kernel: Redzone  ffff8d7f417ae020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:38 zfstest kernel: Redzone  ffff8d7f417ae030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:38 zfstest kernel: Object   ffff8d7f417ae040: 00 00 00 10 9f 01 1e 00 00 00 40 20 89 00 12 00  ..........@ ....
Oct 18 20:08:38 zfstest kernel: Object   ffff8d7f417ae050: 00 00 00 40 89 00 12 00 6b 6b 6b 6b 6b 6b 6b 6b  [email protected]
Oct 18 20:08:38 zfstest kernel: Object   ffff8d7f417ae060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:38 zfstest kernel: Object   ffff8d7f417ae070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:08:38 zfstest kernel: Redzone  ffff8d7f417ae080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:38 zfstest kernel: Padding  ffff8d7f417ae090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:38 zfstest kernel: Padding  ffff8d7f417ae0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:38 zfstest kernel: Padding  ffff8d7f417ae0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:38 zfstest kernel: CPU: 56 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:38 zfstest kernel: Call Trace:
Oct 18 20:08:38 zfstest kernel:  <TASK>
Oct 18 20:08:38 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:08:38 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:08:38 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:08:38 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:08:38 zfstest kernel: ? zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:08:38 zfstest kernel: ? zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:08:38 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:08:38 zfstest kernel: zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:08:38 zfstest kernel: zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:08:38 zfstest kernel: zfs_acl_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:506) zfs
Oct 18 20:08:38 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:08:38 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:08:38 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:08:38 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:08:38 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:08:38 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:08:38 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:08:38 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:08:38 zfstest kernel: zfs_prune (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_vfsops.c:1525) zfs
Oct 18 20:08:38 zfstest kernel: zpl_prune_sb (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zpl_super.c:384) zfs
Oct 18 20:08:38 zfstest kernel: arc_prune_task (/usr/src/linux-headers-6.10.14-xxx/./arch/x86/include/asm/atomic64_64.h:79 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:2913 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:3364 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-instrumented.h:2189 /tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:6080) zfs
Oct 18 20:08:38 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:08:38 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:08:38 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:08:38 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:08:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:38 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:08:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:38 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:08:38 zfstest kernel:  </TASK>
Oct 18 20:08:38 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff8d7f417ae000-0xffff8d7f417ae00f=0xcc
Oct 18 20:08:38 zfstest kernel: FIX kmalloc-64: Object at 0xffff8d7f417ae040 not freed
Oct 18 20:08:41 zfstest kernel: =============================================================================
Oct 18 20:08:41 zfstest kernel: BUG kmalloc-96 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:41 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:41 zfstest kernel: 0xffff8d7ffec02000-0xffff8d7ffec02007 @offset=0. First byte 0x88 instead of 0xcc
Oct 18 20:08:41 zfstest kernel: Slab 0xffffd66887fb0080 objects=64 used=43 fp=0xffff8d7ffec03688 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:41 zfstest kernel: Object 0xffff8d7ffec02008 @offset=8 fp=0xffff8d7ffec02d88
Oct 18 20:08:41 zfstest kernel: Redzone  ffff8d7ffec02000: 88 ae 6b 01 0a 5e 23 a3                          ..k..^#.
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02008: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02018: 01 00 00 00 00 00 00 00 d8 79 fe 73 7f 8d ff ff  .........y.s....
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02028: 00 00 00 00 00 00 00 00 00 41 5a 4c 80 8d ff ff  .........AZL....
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02038: 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  @...............
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02048: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:08:41 zfstest kernel: Object   ffff8d7ffec02058: c0 f7 89 c0 ff ff ff ff 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:08:41 zfstest kernel: Redzone  ffff8d7ffec02068: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:41 zfstest kernel: Padding  ffff8d7ffec02078: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:08:41 zfstest kernel: CPU: 65 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:41 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:41 zfstest kernel: Call Trace:
Oct 18 20:08:41 zfstest kernel:  <TASK>
Oct 18 20:08:41 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:08:41 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:08:41 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:08:41 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:08:41 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:08:41 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:08:41 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:08:41 zfstest kernel: ? zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:08:41 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:08:41 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:08:41 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:08:41 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:08:41 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:08:41 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:08:41 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:08:41 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:08:41 zfstest kernel: zfs_prune (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_vfsops.c:1525) zfs
Oct 18 20:08:41 zfstest kernel: zpl_prune_sb (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zpl_super.c:384) zfs
Oct 18 20:08:41 zfstest kernel: arc_prune_task (/usr/src/linux-headers-6.10.14-xxx/./arch/x86/include/asm/atomic64_64.h:79 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:2913 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:3364 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-instrumented.h:2189 /tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:6080) zfs
Oct 18 20:08:41 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:08:41 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:08:41 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:08:41 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:08:41 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:41 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:08:41 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:41 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:08:41 zfstest kernel:  </TASK>
Oct 18 20:08:41 zfstest kernel: FIX kmalloc-96: Restoring Left Redzone 0xffff8d7ffec02000-0xffff8d7ffec02007=0xcc
Oct 18 20:08:41 zfstest kernel: FIX kmalloc-96: Object at 0xffff8d7ffec02008 not freed
Oct 18 20:08:42 zfstest kernel: =============================================================================
Oct 18 20:08:42 zfstest kernel: BUG kmalloc-512 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:42 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:42 zfstest kernel: 0xffff8d805d128000-0xffff8d805d12800f @offset=0. First byte 0xf9 instead of 0xcc
Oct 18 20:08:42 zfstest kernel: Slab 0xffffd66889744a00 objects=21 used=18 fp=0xffff8d805d12c400 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:42 zfstest kernel: Object 0xffff8d805d128200 @offset=512 fp=0x0000000000000000
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128000: f9 f4 6d b7 29 cb 6b da 48 b1 e5 98 ec 7f 75 e3  ..m.).k.H.....u.
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128040: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128050: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128060: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128070: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128080: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128090: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1280f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128100: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128110: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128120: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128130: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128140: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128150: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128160: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128170: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128180: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128190: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d1281f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128200: 5a 50 2f 00 02 04 18 00 a4 81 00 00 00 00 00 00  ZP/.............
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128210: 40 d9 00 00 00 00 00 00 d9 05 00 00 00 00 00 00  @...............
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128230: 9a 03 00 00 00 00 00 00 04 00 00 00 08 84 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128240: 11 ed 8e 63 00 00 00 00 00 00 00 00 00 00 00 00  ...c............
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128250: 11 ed 8e 63 00 00 00 00 00 00 00 00 00 00 00 00  ...c............
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128260: 69 f8 12 67 00 00 00 00 00 3b 58 08 00 00 00 00  i..g.....;X.....
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128270: 69 f8 12 67 00 00 00 00 00 3b 58 08 00 00 00 00  i..g.....;X.....
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128280: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128290: 03 00 00 00 00 00 00 00 00 00 00 10 9f 01 1e 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282a0: 00 00 40 20 89 00 12 00 00 00 00 40 89 00 12 00  ..@ .......@....
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1282f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128340: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128350: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128360: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128370: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128380: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d128390: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:42 zfstest kernel: Object   ffff8d805d1283f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:08:42 zfstest kernel: Redzone  ffff8d805d128400: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128410: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128420: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128430: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128440: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128450: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128460: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128470: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128480: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128490: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1284f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128500: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128510: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128520: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128530: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128540: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128550: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128560: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128570: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128580: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d128590: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: Padding  ffff8d805d1285f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:42 zfstest kernel: CPU: 103 PID: 1503 Comm: dbu_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:42 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:42 zfstest kernel: Call Trace:
Oct 18 20:08:42 zfstest kernel:  <TASK>
Oct 18 20:08:42 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:08:42 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:08:42 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:08:42 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:08:42 zfstest kernel: ? dbuf_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:754) zfs
Oct 18 20:08:42 zfstest kernel: ? dbuf_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:754) zfs
Oct 18 20:08:42 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:08:42 zfstest kernel: dbuf_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:754) zfs
Oct 18 20:08:42 zfstest kernel: dnode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:676) zfs
Oct 18 20:08:42 zfstest kernel: dnode_buf_evict_async (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1351) zfs
Oct 18 20:08:42 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:08:42 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:08:42 zfstest kernel: ? dnode_create (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1947) zfs
Oct 18 20:08:42 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:08:42 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:08:42 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:42 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:08:42 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:42 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:08:42 zfstest kernel:  </TASK>
Oct 18 20:08:42 zfstest kernel: FIX kmalloc-512: Restoring Left Redzone 0xffff8d805d128000-0xffff8d805d12800f=0xcc
Oct 18 20:08:42 zfstest kernel: FIX kmalloc-512: Object at 0xffff8d805d128200 not freed
Oct 18 20:08:43 zfstest kernel: =============================================================================
Oct 18 20:08:43 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:43 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:43 zfstest kernel: 0xffff8d7f9e5fa000-0xffff8d7f9e5fa00f @offset=0. First byte 0x49 instead of 0xcc
Oct 18 20:08:43 zfstest kernel: Slab 0xffffd66886797e80 objects=42 used=30 fp=0xffff8d7f9e5fa100 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:43 zfstest kernel: Object 0xffff8d7f9e5fa040 @offset=64 fp=0xffff8d7f9e5fbc00
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7f9e5fa000: 49 dc 09 93 87 31 d2 93 0e 0d dd 6d 72 61 72 36  I....1.....mrar6
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7f9e5fa010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7f9e5fa020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7f9e5fa030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7f9e5fa040: 00 00 00 10 9f 01 1e 00 00 00 40 20 88 00 12 00  ..........@ ....
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7f9e5fa050: 00 00 00 40 88 00 12 00 6b 6b 6b 6b 6b 6b 6b 6b  [email protected]
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7f9e5fa060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7f9e5fa070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7f9e5fa080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:43 zfstest kernel: Padding  ffff8d7f9e5fa090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:43 zfstest kernel: Padding  ffff8d7f9e5fa0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:43 zfstest kernel: Padding  ffff8d7f9e5fa0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:43 zfstest kernel: CPU: 23 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:43 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:43 zfstest kernel: Call Trace:
Oct 18 20:08:43 zfstest kernel:  <TASK>
Oct 18 20:08:43 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:08:43 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:08:43 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:08:43 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:08:43 zfstest kernel: ? zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:08:43 zfstest kernel: ? zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:08:43 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:08:43 zfstest kernel: zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:08:43 zfstest kernel: zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:08:43 zfstest kernel: zfs_acl_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:506) zfs
Oct 18 20:08:43 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:08:43 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:08:43 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:08:43 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:08:43 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:08:43 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:08:43 zfstest kernel: zfs_prune (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_vfsops.c:1525) zfs
Oct 18 20:08:43 zfstest kernel: zpl_prune_sb (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zpl_super.c:384) zfs
Oct 18 20:08:43 zfstest kernel: arc_prune_task (/usr/src/linux-headers-6.10.14-xxx/./arch/x86/include/asm/atomic64_64.h:79 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:2913 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:3364 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-instrumented.h:2189 /tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:6080) zfs
Oct 18 20:08:43 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:08:43 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:08:43 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:08:43 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:08:43 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:43 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:08:43 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:43 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:08:43 zfstest kernel:  </TASK>
Oct 18 20:08:43 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff8d7f9e5fa000-0xffff8d7f9e5fa00f=0xcc
Oct 18 20:08:43 zfstest kernel: FIX kmalloc-64: Object at 0xffff8d7f9e5fa040 not freed
Oct 18 20:08:43 zfstest kernel: =============================================================================
Oct 18 20:08:43 zfstest kernel: BUG dnode_t (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:43 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:43 zfstest kernel: 0xffff8d7e74400000-0xffff8d7e74400007 @offset=0. First byte 0x3e instead of 0xcc
Oct 18 20:08:43 zfstest kernel: Slab 0xffffd66881d10000 objects=32 used=32 fp=0x0000000000000000 flags=0xfffffc0000040(head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:08:43 zfstest kernel: Object 0xffff8d7e74400008 @offset=8 fp=0xffff8d7e74405360
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7e74400000: 3e 88 19 c6 ac f6 55 34                          >.....U4
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400008: 49 55 13 df 22 f5 59 cc c1 b3 6b fc 7f 8d ff ff  IU..".Y...k.....
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400018: 00 00 00 00 00 00 00 00 20 00 40 74 7e 8d ff ff  ........ .@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400028: 20 00 40 74 7e 8d ff ff 00 00 00 00 00 00 00 00   .@t~...........
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400038: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400048: 01 b8 c6 a1 7f 8d ff ff 02 f7 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400058: 98 50 aa 62 7e 8d ff ff b0 c1 88 47 7e 8d ff ff  .P.b~......G~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400068: 00 04 a0 3d 80 8d ff ff 14 00 00 00 b0 00 2c 01  ...=..........,.
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400078: 00 00 01 11 09 00 01 00 00 02 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400088: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400098: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000a8: 00 6b 00 00 00 00 00 00 00 00 6b 6b 00 00 00 00  .k........kk....
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000b8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000e8: 00 00 00 00 6b 6b 6b 6b 00 01 00 00 00 00 ad de  ....kkkk........
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744000f8: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400108: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400118: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400128: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400138: 00 00 00 00 00 00 00 00 40 01 40 74 7e 8d ff ff  ........@.@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400148: 40 01 40 74 7e 8d ff ff 00 00 00 00 6b 6b 6b 6b  @.@t~.......kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400158: 00 00 00 00 00 00 00 00 98 01 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400168: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400178: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400188: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400198: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001a8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001b8: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001c8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001d8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744001f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400208: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400218: 00 00 00 00 00 00 00 00 f5 45 65 34 6b 6b 6b 6b  .........Ee4kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400228: 00 00 00 00 6b 6b 6b 6b 30 02 40 74 7e 8d ff ff  ....kkkk0.@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400238: 30 02 40 74 7e 8d ff ff 00 00 00 00 6b 6b 6b 6b  0.@t~.......kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400248: 48 02 40 74 7e 8d ff ff 48 02 40 74 7e 8d ff ff  H.@t~...H.@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400258: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400268: f5 45 65 34 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b  .Ee4kkkk....kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400278: 78 02 40 74 7e 8d ff ff 78 02 40 74 7e 8d ff ff  x.@t~...x.@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400288: 00 00 00 00 6b 6b 6b 6b 90 02 40 74 7e 8d ff ff  ....kkkk..@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400298: 90 02 40 74 7e 8d ff ff 00 00 00 00 00 00 00 00  ..@t~...........
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002a8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002b8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002d8: 00 00 00 00 00 00 00 00 e0 02 40 74 7e 8d ff ff  ..........@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002e8: e0 02 40 74 7e 8d ff ff 00 00 00 00 6b 6b 6b 6b  ..@t~.......kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744002f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400308: 20 b5 51 c0 ff ff ff ff 40 00 00 00 00 00 00 00   .Q.....@.......
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400318: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400328: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400338: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400348: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400358: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400368: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400378: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400388: 00 00 00 00 00 00 00 00 90 03 40 74 7e 8d ff ff  ..........@t~...
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e74400398: 90 03 40 74 7e 8d ff ff 00 00 00 00 6b 6b 6b 6b  ..@t~.......kkkk
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744003a8: 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744003b8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744003c8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:08:43 zfstest kernel: Object   ffff8d7e744003d8: 00 00 00 00 6b 6b 6b a5                          ....kkk.
Oct 18 20:08:43 zfstest kernel: Redzone  ffff8d7e744003e0: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:43 zfstest kernel: Padding  ffff8d7e744003f0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:08:43 zfstest kernel: CPU: 109 PID: 1503 Comm: dbu_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:43 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:43 zfstest kernel: Call Trace:
Oct 18 20:08:43 zfstest kernel:  <TASK>
Oct 18 20:08:43 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:08:43 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:08:43 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:08:43 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:08:43 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:08:43 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:08:43 zfstest kernel: kmem_cache_free (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4523)
Oct 18 20:08:43 zfstest kernel: spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:08:43 zfstest kernel: dnode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:648 (discriminator 1)) zfs
Oct 18 20:08:43 zfstest kernel: dnode_buf_evict_async (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1351) zfs
Oct 18 20:08:43 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:08:43 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:08:43 zfstest kernel: ? dnode_create (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1947) zfs
Oct 18 20:08:43 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:08:43 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:08:43 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:43 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:08:43 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:43 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:08:43 zfstest kernel:  </TASK>
Oct 18 20:08:43 zfstest kernel: FIX dnode_t: Restoring Left Redzone 0xffff8d7e74400000-0xffff8d7e74400007=0xcc
Oct 18 20:08:43 zfstest kernel: FIX dnode_t: Object at 0xffff8d7e74400008 not freed
Oct 18 20:08:44 zfstest kernel: =============================================================================
Oct 18 20:08:44 zfstest kernel: BUG kmalloc-32 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:44 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:44 zfstest kernel: 0xffff8d7ffb851000-0xffff8d7ffb85100f @offset=0. First byte 0x31 instead of 0xcc
Oct 18 20:08:44 zfstest kernel: Slab 0xffffd66887ee1440 objects=42 used=36 fp=0xffff8d7ffb851500 flags=0x17ffffc0000200(workingset|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:44 zfstest kernel: Object 0xffff8d7ffb851020 @offset=32 fp=0x0000000000000000
Oct 18 20:08:44 zfstest kernel: Redzone  ffff8d7ffb851000: 31 08 21 24 54 78 da 0d 4c b0 f1 fd b6 a4 ea b3  1.!$Tx..L.......
Oct 18 20:08:44 zfstest kernel: Redzone  ffff8d7ffb851010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:44 zfstest kernel: Object   ffff8d7ffb851020: 00 00 00 00 01 00 00 00 80 b4 f0 bd 7f 8d ff ff  ................
Oct 18 20:08:44 zfstest kernel: Object   ffff8d7ffb851030: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:08:44 zfstest kernel: Redzone  ffff8d7ffb851040: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:44 zfstest kernel: Padding  ffff8d7ffb851050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:44 zfstest kernel: CPU: 23 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:44 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:44 zfstest kernel: Call Trace:
Oct 18 20:08:44 zfstest kernel:  <TASK>
Oct 18 20:08:44 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:08:44 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:08:44 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:08:44 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:08:44 zfstest kernel: ? nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:08:44 zfstest kernel: ? nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:08:44 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:08:44 zfstest kernel: ? _raw_spin_unlock_irqrestore (./include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:194)
Oct 18 20:08:44 zfstest kernel: nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:08:44 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:08:44 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:08:44 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:08:44 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:08:44 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:08:44 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:08:44 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:08:44 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:08:44 zfstest kernel: zfs_prune (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_vfsops.c:1525) zfs
Oct 18 20:08:44 zfstest kernel: zpl_prune_sb (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zpl_super.c:384) zfs
Oct 18 20:08:44 zfstest kernel: arc_prune_task (/usr/src/linux-headers-6.10.14-xxx/./arch/x86/include/asm/atomic64_64.h:79 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:2913 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:3364 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-instrumented.h:2189 /tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:6080) zfs
Oct 18 20:08:44 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:08:44 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:08:44 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:08:44 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:08:44 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:44 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:08:44 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:44 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:08:44 zfstest kernel:  </TASK>
Oct 18 20:08:44 zfstest kernel: FIX kmalloc-32: Restoring Left Redzone 0xffff8d7ffb851000-0xffff8d7ffb85100f=0xcc
Oct 18 20:08:44 zfstest kernel: FIX kmalloc-32: Object at 0xffff8d7ffb851020 not freed
Oct 18 20:08:46 zfstest kernel: =============================================================================
Oct 18 20:08:46 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:46 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:46 zfstest kernel: 0xffff8d806862c000-0xffff8d806862c00f @offset=0. First byte 0x5 instead of 0xcc
Oct 18 20:08:46 zfstest kernel: Slab 0xffffd66889a18b00 objects=42 used=24 fp=0xffff8d806862d600 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:46 zfstest kernel: Object 0xffff8d806862c040 @offset=64 fp=0x0000000000000000
Oct 18 20:08:46 zfstest kernel: Redzone  ffff8d806862c000: 05 85 17 77 57 a1 e4 98 9a f6 d0 7f 5b b9 1d f9  ...wW.......[...
Oct 18 20:08:46 zfstest kernel: Redzone  ffff8d806862c010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:46 zfstest kernel: Redzone  ffff8d806862c020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:46 zfstest kernel: Redzone  ffff8d806862c030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:46 zfstest kernel: Object   ffff8d806862c040: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:08:46 zfstest kernel: Object   ffff8d806862c050: 80 9a cf c4 7f 8d ff ff 80 9a cf c4 7f 8d ff ff  ................
Oct 18 20:08:46 zfstest kernel: Object   ffff8d806862c060: 30 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00  0...............
Oct 18 20:08:46 zfstest kernel: Object   ffff8d806862c070: 03 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00  ................
Oct 18 20:08:46 zfstest kernel: Redzone  ffff8d806862c080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:46 zfstest kernel: Padding  ffff8d806862c090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:46 zfstest kernel: Padding  ffff8d806862c0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:46 zfstest kernel: Padding  ffff8d806862c0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:46 zfstest kernel: CPU: 30 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:46 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:46 zfstest kernel: Call Trace:
Oct 18 20:08:46 zfstest kernel:  <TASK>
Oct 18 20:08:46 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:08:46 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:08:46 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:08:46 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:08:46 zfstest kernel: ? zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:08:46 zfstest kernel: ? zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:08:46 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:08:46 zfstest kernel: zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:08:46 zfstest kernel: zfs_acl_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:506) zfs
Oct 18 20:08:46 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:08:46 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:08:46 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:08:46 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:08:46 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:08:46 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:08:46 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:08:46 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:08:46 zfstest kernel: zfs_prune (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_vfsops.c:1525) zfs
Oct 18 20:08:46 zfstest kernel: zpl_prune_sb (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zpl_super.c:384) zfs
Oct 18 20:08:46 zfstest kernel: arc_prune_task (/usr/src/linux-headers-6.10.14-xxx/./arch/x86/include/asm/atomic64_64.h:79 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:2913 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:3364 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-instrumented.h:2189 /tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:6080) zfs
Oct 18 20:08:46 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:08:46 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:08:46 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:08:46 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:08:46 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:46 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:08:46 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:46 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:08:46 zfstest kernel:  </TASK>
Oct 18 20:08:46 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff8d806862c000-0xffff8d806862c00f=0xcc
Oct 18 20:08:46 zfstest kernel: FIX kmalloc-64: Object at 0xffff8d806862c040 not freed
Oct 18 20:08:47 zfstest kernel: =============================================================================
Oct 18 20:08:47 zfstest kernel: BUG kmalloc-96 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:47 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:47 zfstest kernel: 0xffff8d7fd8982000-0xffff8d7fd8982007 @offset=0. First byte 0xfe instead of 0xcc
Oct 18 20:08:47 zfstest kernel: Slab 0xffffd66887626080 objects=64 used=47 fp=0xffff8d7fd8982f88 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:47 zfstest kernel: Object 0xffff8d7fd8982008 @offset=8 fp=0xffff8d7fd8982108
Oct 18 20:08:47 zfstest kernel: Redzone  ffff8d7fd8982000: fe a4 c1 99 23 d9 28 30                          ....#.(0
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982008: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982018: 01 00 00 00 00 00 00 00 18 2d cd 88 7f 8d ff ff  .........-......
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982028: 00 00 00 00 00 00 00 00 80 8e d9 3a 80 8d ff ff  ...........:....
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982038: 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  @...............
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982048: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:08:47 zfstest kernel: Object   ffff8d7fd8982058: c0 f7 89 c0 ff ff ff ff 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:08:47 zfstest kernel: Redzone  ffff8d7fd8982068: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:47 zfstest kernel: Padding  ffff8d7fd8982078: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:08:47 zfstest kernel: CPU: 53 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:47 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:47 zfstest kernel: Call Trace:
Oct 18 20:08:47 zfstest kernel:  <TASK>
Oct 18 20:08:47 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:08:47 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:08:47 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:08:47 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:08:47 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:08:47 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:08:47 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:08:47 zfstest kernel: ? zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:08:47 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:08:47 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:08:47 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:08:47 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:08:47 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:08:47 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:08:47 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:08:47 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:08:47 zfstest kernel: zfs_prune (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_vfsops.c:1525) zfs
Oct 18 20:08:47 zfstest kernel: zpl_prune_sb (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zpl_super.c:384) zfs
Oct 18 20:08:47 zfstest kernel: arc_prune_task (/usr/src/linux-headers-6.10.14-xxx/./arch/x86/include/asm/atomic64_64.h:79 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:2913 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:3364 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-instrumented.h:2189 /tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:6080) zfs
Oct 18 20:08:47 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:08:47 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:08:47 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:08:47 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:08:47 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:47 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:08:47 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:47 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:08:47 zfstest kernel:  </TASK>
Oct 18 20:08:47 zfstest kernel: FIX kmalloc-96: Restoring Left Redzone 0xffff8d7fd8982000-0xffff8d7fd8982007=0xcc
Oct 18 20:08:47 zfstest kernel: FIX kmalloc-96: Object at 0xffff8d7fd8982008 not freed
Oct 18 20:08:51 zfstest kernel: =============================================================================
Oct 18 20:08:51 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:51 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:51 zfstest kernel: 0xffff8d7feee7a000-0xffff8d7feee7a00f @offset=0. First byte 0xea instead of 0xcc
Oct 18 20:08:51 zfstest kernel: Slab 0xffffd66887bb9e80 objects=42 used=34 fp=0xffff8d7feee7ae80 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:08:51 zfstest kernel: Object 0xffff8d7feee7a040 @offset=64 fp=0x0000000000000000
Oct 18 20:08:51 zfstest kernel: Redzone  ffff8d7feee7a000: ea 20 d5 50 0a 52 f5 a0 75 eb d7 05 32 81 a0 7c  . .P.R..u...2..|
Oct 18 20:08:51 zfstest kernel: Redzone  ffff8d7feee7a010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:51 zfstest kernel: Redzone  ffff8d7feee7a020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:51 zfstest kernel: Redzone  ffff8d7feee7a030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:08:51 zfstest kernel: Object   ffff8d7feee7a040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:51 zfstest kernel: Object   ffff8d7feee7a050: 00 00 00 00 00 00 00 00 50 a3 6a c0 ff ff ff ff  ........P.j.....
Oct 18 20:08:51 zfstest kernel: Object   ffff8d7feee7a060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:51 zfstest kernel: Object   ffff8d7feee7a070: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:08:51 zfstest kernel: Redzone  ffff8d7feee7a080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:51 zfstest kernel: Padding  ffff8d7feee7a090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:51 zfstest kernel: Padding  ffff8d7feee7a0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:51 zfstest kernel: Padding  ffff8d7feee7a0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:08:51 zfstest kernel: CPU: 109 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:51 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:51 zfstest kernel: Call Trace:
Oct 18 20:08:51 zfstest kernel:  <TASK>
Oct 18 20:08:51 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:08:51 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:08:51 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:08:51 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:08:51 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:08:51 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:08:51 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:08:51 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:08:51 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:08:51 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:08:51 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:08:51 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:08:51 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:08:51 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:08:51 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:08:51 zfstest kernel: zfs_prune (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_vfsops.c:1525) zfs
Oct 18 20:08:51 zfstest kernel: zpl_prune_sb (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zpl_super.c:384) zfs
Oct 18 20:08:51 zfstest kernel: arc_prune_task (/usr/src/linux-headers-6.10.14-xxx/./arch/x86/include/asm/atomic64_64.h:79 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:2913 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:3364 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-instrumented.h:2189 /tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:6080) zfs
Oct 18 20:08:51 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:08:51 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:08:51 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:08:51 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:08:51 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:51 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:08:51 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:51 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:08:51 zfstest kernel:  </TASK>
Oct 18 20:08:51 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff8d7feee7a000-0xffff8d7feee7a00f=0xcc
Oct 18 20:08:51 zfstest kernel: FIX kmalloc-64: Object at 0xffff8d7feee7a040 not freed
Oct 18 20:08:52 zfstest kernel: =============================================================================
Oct 18 20:08:52 zfstest kernel: BUG lsm_inode_cache (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:08:52 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:08:52 zfstest kernel: 0xffff8d7e40122000-0xffff8d7e40122007 @offset=0. First byte 0x5f instead of 0xcc
Oct 18 20:08:52 zfstest kernel: Slab 0xffffd66881004880 objects=42 used=11 fp=0xffff8d7e40122d28 flags=0xfffffc0000200(workingset|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:08:52 zfstest kernel: Object 0xffff8d7e40122008 @offset=8 fp=0xffff8d7e40122cc8
Oct 18 20:08:52 zfstest kernel: Redzone  ffff8d7e40122000: 5f 33 02 f5 75 d7 04 e6                          _3..u...
Oct 18 20:08:52 zfstest kernel: Object   ffff8d7e40122008: 88 fa 5c 1c 7e 8d ff ff 00 00 00 00 00 00 00 00  ...~...........
Oct 18 20:08:52 zfstest kernel: Object   ffff8d7e40122018: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:52 zfstest kernel: Object   ffff8d7e40122028: 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:52 zfstest kernel: Object   ffff8d7e40122038: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:08:52 zfstest kernel: Redzone  ffff8d7e40122048: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:08:52 zfstest kernel: Padding  ffff8d7e40122058: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:08:52 zfstest kernel: CPU: 92 PID: 1500 Comm: arc_prune Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:08:52 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:08:52 zfstest kernel: Call Trace:
Oct 18 20:08:52 zfstest kernel:  <IRQ>
Oct 18 20:08:52 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:08:52 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:08:52 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:08:52 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:08:52 zfstest kernel: ? rcu_core (kernel/rcu/tree.c:2542 kernel/rcu/tree.c:2809)
Oct 18 20:08:52 zfstest kernel: ? rcu_core (kernel/rcu/tree.c:2542 kernel/rcu/tree.c:2809)
Oct 18 20:08:52 zfstest kernel: kmem_cache_free (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4523)
Oct 18 20:08:52 zfstest kernel: ? rcu_core (./include/linux/bottom_half.h:33 kernel/rcu/tree.c:2554 kernel/rcu/tree.c:2809)
Oct 18 20:08:52 zfstest kernel: rcu_core (kernel/rcu/tree.c:2542 kernel/rcu/tree.c:2809)
Oct 18 20:08:52 zfstest kernel: ? kvm_clock_read (./arch/x86/include/asm/preempt.h:94 arch/x86/kernel/kvmclock.c:80)
Oct 18 20:08:52 zfstest kernel: handle_softirqs (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:207 ./include/trace/events/irq.h:142 kernel/softirq.c:555)
Oct 18 20:08:52 zfstest kernel: irq_exit_rcu (kernel/softirq.c:589 kernel/softirq.c:428 kernel/softirq.c:637 kernel/softirq.c:649)
Oct 18 20:08:52 zfstest kernel: sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1043 arch/x86/kernel/apic/apic.c:1043)
Oct 18 20:08:52 zfstest kernel:  </IRQ>
Oct 18 20:08:52 zfstest kernel:  <TASK>
Oct 18 20:08:52 zfstest kernel: asm_sysvec_apic_timer_interrupt (./arch/x86/include/asm/idtentry.h:702)
Oct 18 20:08:52 zfstest kernel: RIP: 0010:_raw_spin_unlock_irqrestore (./include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:194)
Oct 18 20:08:52 zfstest kernel: Code: d8 0d 6c 85 c0 74 01 c3 e8 d4 cb 2a ff c3 0f 1f 00 0f 1f 44 00 00 c6 07 00 0f 1f 00 f7 c6 00 02 00 00 74 06 fb 0f 1f 44 00 00 <bf> 01 00 00 00 e8 ed 03 39 ff 65 8b 05 9e d8 0d 6c 85 c0 74 01 c3
All code
========
   0: d8 0d 6c 85 c0 74     fmuls  0x74c0856c(%rip)        # 0x74c08572
   6: 01 c3                 add    %eax,%ebx
   8: e8 d4 cb 2a ff        call   0xffffffffff2acbe1
   d: c3                    ret
   e: 0f 1f 00              nopl   (%rax)
  11: 0f 1f 44 00 00        nopl   0x0(%rax,%rax,1)
  16: c6 07 00              movb   $0x0,(%rdi)
  19: 0f 1f 00              nopl   (%rax)
  1c: f7 c6 00 02 00 00     test   $0x200,%esi
  22: 74 06                 je     0x2a
  24: fb                    sti
  25: 0f 1f 44 00 00        nopl   0x0(%rax,%rax,1)
  2a:* bf 01 00 00 00        mov    $0x1,%edi  <-- trapping instruction
  2f: e8 ed 03 39 ff        call   0xffffffffff390421
  34: 65 8b 05 9e d8 0d 6c  mov    %gs:0x6c0dd89e(%rip),%eax        # 0x6c0dd8d9
  3b: 85 c0                 test   %eax,%eax
  3d: 74 01                 je     0x40
  3f: c3                    ret

Code starting with the faulting instruction
===========================================
   0: bf 01 00 00 00        mov    $0x1,%edi
   5: e8 ed 03 39 ff        call   0xffffffffff3903f7
   a: 65 8b 05 9e d8 0d 6c  mov    %gs:0x6c0dd89e(%rip),%eax        # 0x6c0dd8af
  11: 85 c0                 test   %eax,%eax
  13: 74 01                 je     0x16
  15: c3                    ret
Oct 18 20:08:52 zfstest kernel: RSP: 0018:ffffa049049cfac8 EFLAGS: 00000206
Oct 18 20:08:52 zfstest kernel: RAX: ffff8d7fe443c580 RBX: ffff8d7fe443c580 RCX: 7643159d32c7016e
Oct 18 20:08:52 zfstest kernel: RDX: ffff8d7fe443f6b8 RSI: 0000000000000246 RDI: ffff8d7f15fbe640
Oct 18 20:08:52 zfstest kernel: RBP: ffffa049049cfb20 R08: ffff8d7fe443c688 R09: 0000000000000000
Oct 18 20:08:52 zfstest kernel: R10: ffff8d7fe443c678 R11: 0000000000000000 R12: ffffd66887910f00
Oct 18 20:08:52 zfstest kernel: R13: ffffd66887910f00 R14: ffff8d7f20fce980 R15: ffffffffc03f7e2b
Oct 18 20:08:52 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:08:52 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:08:52 zfstest kernel: kmem_cache_free (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4523)
Oct 18 20:08:52 zfstest kernel: ? _raw_spin_lock (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:134 kernel/locking/spinlock.c:154)
Oct 18 20:08:52 zfstest kernel: spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:08:52 zfstest kernel: zfs_zinactive (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:1391) zfs
Oct 18 20:08:52 zfstest kernel: zfs_inactive (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_vnops_os.c:3959) zfs
Oct 18 20:08:52 zfstest kernel: ? unmap_mapping_pages (mm/memory.c:3785)
Oct 18 20:08:52 zfstest kernel: zpl_evict_inode (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zpl_super.c:104) zfs
Oct 18 20:08:52 zfstest kernel: evict (fs/inode.c:711)
Oct 18 20:08:52 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:08:52 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:08:52 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:08:52 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:08:52 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:08:52 zfstest kernel: zfs_prune (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_vfsops.c:1525) zfs
Oct 18 20:08:52 zfstest kernel: zpl_prune_sb (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zpl_super.c:384) zfs
Oct 18 20:08:52 zfstest kernel: arc_prune_task (/usr/src/linux-headers-6.10.14-xxx/./arch/x86/include/asm/atomic64_64.h:79 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:2913 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-arch-fallback.h:3364 /usr/src/linux-headers-6.10.14-xxx/./include/linux/atomic/atomic-instrumented.h:2189 /tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:6080) zfs
Oct 18 20:08:52 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:08:52 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:08:52 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:08:52 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:08:52 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:52 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:08:52 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:08:52 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:08:52 zfstest kernel:  </TASK>
Oct 18 20:08:52 zfstest kernel: FIX lsm_inode_cache: Restoring Left Redzone 0xffff8d7e40122000-0xffff8d7e40122007=0xcc
Oct 18 20:08:52 zfstest kernel: FIX lsm_inode_cache: Object at 0xffff8d7e40122008 not freed

Trace 2 decoded

Oct 18 20:15:35 zfstest kernel: =============================================================================
Oct 18 20:15:35 zfstest kernel: BUG kmalloc-512 (Tainted: P           O      ): Left Redzone overwritten
Oct 18 20:15:35 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:35 zfstest kernel: 0xffff9bec2d48f000-0xffff9bec2d48f00f @offset=0. First byte 0xa0 instead of 0xcc
Oct 18 20:15:35 zfstest kernel: Slab 0xffffe87080b523c0 objects=2 used=2 fp=0x0000000000000000 flags=0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:35 zfstest kernel: Object 0xffff9bec2d48f200 @offset=512 fp=0xffff9bec2d48f800
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f000: a0 f8 e7 f7 68 19 9d bc 91 8a 10 da d5 b4 bb f8  ....h...........
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f040: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f050: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f060: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f070: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f080: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f090: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f0f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f100: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f110: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f120: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f130: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f140: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f150: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f160: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f170: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f180: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f190: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f1f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f200: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f210: 70 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00  p...............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f220: 48 4d c9 02 ec 9b ff ff 00 01 00 00 00 00 ad de  HM..............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f230: 22 01 00 00 00 00 ad de b8 a7 72 62 ec 9b ff ff  ".........rb....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f290: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2d0: 00 b2 35 30 ec 9b ff ff 00 00 00 00 00 00 00 00  ..50............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f2f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f390: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b  ........kkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec2d48f3f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec2d48f400: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f410: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f420: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f430: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f440: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f450: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f460: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f470: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f480: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f490: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f4f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f500: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f510: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f520: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f530: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f540: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f550: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f560: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f570: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f580: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f590: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec2d48f5f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: CPU: 82 PID: 1917 Comm: dp_sync_taskq Tainted: P           O       6.10.14-xxx #1
Oct 18 20:15:35 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:35 zfstest kernel: Call Trace:
Oct 18 20:15:35 zfstest kernel:  <TASK>
Oct 18 20:15:35 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:35 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:35 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:35 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:35 zfstest kernel: ? dbuf_sync_leaf (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:2319) zfs
Oct 18 20:15:35 zfstest kernel: ? dbuf_sync_leaf (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:2319) zfs
Oct 18 20:15:35 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:35 zfstest kernel: ? zio_taskq_dispatch (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/zio.c:2042) zfs
Oct 18 20:15:35 zfstest kernel: ? zio_nowait (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/zio.c:2333 /tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/zio.c:2433) zfs
Oct 18 20:15:35 zfstest kernel: dbuf_sync_leaf (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:2319) zfs
Oct 18 20:15:35 zfstest kernel: ? zpl_get_file_info (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/zfs_quota.c:91 (discriminator 3)) zfs
Oct 18 20:15:35 zfstest kernel: ? preempt_count_add (./include/linux/ftrace.h:975 kernel/sched/core.c:5850 kernel/sched/core.c:5847 kernel/sched/core.c:5875)
Oct 18 20:15:35 zfstest kernel: ? _raw_spin_lock (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:134 kernel/locking/spinlock.c:154)
Oct 18 20:15:35 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:35 zfstest kernel: ? dmu_objset_userquota_get_ids (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dmu_objset.c:2278) zfs
Oct 18 20:15:35 zfstest kernel: dbuf_sync_list (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:4625 (discriminator 1)) zfs
Oct 18 20:15:35 zfstest kernel: dnode_sync (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode_sync.c:873) zfs
Oct 18 20:15:35 zfstest kernel: ? free_to_partial_list (mm/slub.c:3292 mm/slub.c:4197)
Oct 18 20:15:35 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:15:35 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:15:35 zfstest kernel: ? _raw_spin_unlock_irqrestore (./include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:194)
Oct 18 20:15:35 zfstest kernel: ? preempt_count_add (./include/linux/ftrace.h:975 kernel/sched/core.c:5850 kernel/sched/core.c:5847 kernel/sched/core.c:5875)
Oct 18 20:15:35 zfstest kernel: sync_dnodes_task (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dmu_objset.c:2097) zfs
Oct 18 20:15:35 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:15:35 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:15:35 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:15:35 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:35 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:35 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:35 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:35 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:35 zfstest kernel:  </TASK>
Oct 18 20:15:35 zfstest kernel: FIX kmalloc-512: Restoring Left Redzone 0xffff9bec2d48f000-0xffff9bec2d48f00f=0xcc
Oct 18 20:15:35 zfstest kernel: FIX kmalloc-512: Object at 0xffff9bec2d48f200 not freed
Oct 18 20:15:35 zfstest kernel: =============================================================================
Oct 18 20:15:35 zfstest kernel: BUG kmalloc-512 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:35 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:35 zfstest kernel: 0xffff9bec3150f000-0xffff9bec3150f00f @offset=0. First byte 0xe8 instead of 0xcc
Oct 18 20:15:35 zfstest kernel: Slab 0xffffe87080c543c0 objects=2 used=2 fp=0x0000000000000000 flags=0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:35 zfstest kernel: Object 0xffff9bec3150f200 @offset=512 fp=0x0000000000000000
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f000: e8 4a 9e 9d 18 08 18 88 ab 25 b8 b1 f9 dd f3 fb  .J.......%......
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f040: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f050: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f060: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f070: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f080: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f090: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f0f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f100: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f110: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f120: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f130: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f140: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f150: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f160: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f170: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f180: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f190: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f1f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f210: 10 f2 50 31 ec 9b ff ff 10 f2 50 31 ec 9b ff ff  ..P1......P1....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f240: 19 00 00 00 00 00 00 00 48 f2 50 31 ec 9b ff ff  ........H.P1....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f250: 48 f2 50 31 ec 9b ff ff 00 00 00 00 00 00 00 00  H.P1............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f280: 00 00 00 00 00 00 00 00 70 bb 5e c0 ff ff ff ff  ........p.^.....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f290: 00 00 00 00 00 00 00 00 00 88 f9 85 ed 9b ff ff  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2a0: ff 82 01 00 00 00 00 00 f8 87 61 3b ec 9b ff ff  ..........a;....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2b0: a8 bb 16 59 ec 9b ff ff 00 00 00 00 00 00 00 00  ...Y............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2d0: d0 f2 50 31 ec 9b ff ff d0 f2 50 31 ec 9b ff ff  ..P1......P1....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2e0: 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f2f0: bf 8b fe 6b 00 00 00 00 04 00 07 00 04 00 00 00  ...k............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f300: 90 b7 5e c0 ff ff ff ff c0 b7 5e c0 ff ff ff ff  ..^.......^.....
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f310: 08 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f320: 3e 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  >...............
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f340: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b  ........kkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f350: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f360: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f370: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f380: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f390: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:35 zfstest kernel: Object   ffff9bec3150f3f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:15:35 zfstest kernel: Redzone  ffff9bec3150f400: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f410: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f420: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f430: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f440: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f450: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f460: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f470: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f480: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f490: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f4f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f500: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f510: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f520: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f530: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f540: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f550: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f560: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f570: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f580: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f590: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: Padding  ffff9bec3150f5f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:35 zfstest kernel: CPU: 86 PID: 2814 Comm: z_wr_iss Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:35 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:35 zfstest kernel: Call Trace:
Oct 18 20:15:35 zfstest kernel:  <TASK>
Oct 18 20:15:35 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:35 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:35 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:35 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:35 zfstest kernel: ? dbuf_evict_user.isra.0+0x31/0x60 zfs
Oct 18 20:15:35 zfstest kernel: ? dbuf_evict_user.isra.0+0x31/0x60 zfs
Oct 18 20:15:35 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:35 zfstest kernel: dbuf_evict_user.isra.0+0x31/0x60 zfs
Oct 18 20:15:35 zfstest kernel: dbuf_clear_data (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:1219) zfs
Oct 18 20:15:35 zfstest kernel: dbuf_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:3027) zfs
Oct 18 20:15:35 zfstest kernel: dbuf_write_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:4862) zfs
Oct 18 20:15:35 zfstest kernel: arc_write_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:6595) zfs
Oct 18 20:15:35 zfstest kernel: zio_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/zio.c:4996) zfs
Oct 18 20:15:35 zfstest kernel: ? zio_ready (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/zio.c:4576) zfs
Oct 18 20:15:35 zfstest kernel: zio_execute (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/zio.c:2346 /tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/zio.c:2257) zfs
Oct 18 20:15:35 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:15:35 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:15:35 zfstest kernel: ? zio_reexecute (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/zio.c:4595) zfs
Oct 18 20:15:35 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:15:35 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:35 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:35 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:35 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:35 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:35 zfstest kernel:  </TASK>
Oct 18 20:15:35 zfstest kernel: FIX kmalloc-512: Restoring Left Redzone 0xffff9bec3150f000-0xffff9bec3150f00f=0xcc
Oct 18 20:15:35 zfstest kernel: FIX kmalloc-512: Object at 0xffff9bec3150f200 not freed
Oct 18 20:15:35 zfstest dockerd[2741]: time="2024-10-18T20:15:35.677731588-04:00" level=info msg="Download failed, retrying (1/5): local error: tls: bad record MAC"
Oct 18 20:15:36 zfstest kernel: =============================================================================
Oct 18 20:15:36 zfstest kernel: BUG kmalloc-96 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:36 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:36 zfstest kernel: 0xffff9bec159d6000-0xffff9bec159d6007 @offset=0. First byte 0xd instead of 0xcc
Oct 18 20:15:36 zfstest kernel: Slab 0xffffe87080567580 objects=64 used=64 fp=0x0000000000000000 flags=0xfffffc0000040(head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:36 zfstest kernel: Object 0xffff9bec159d6008 @offset=8 fp=0xffff9bec159d6508
Oct 18 20:15:36 zfstest kernel: Redzone  ffff9bec159d6000: 0d ee 5b 6c 6c 4c ae 9f                          ..[llL..
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6008: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6018: 01 00 00 00 00 00 00 00 98 c5 93 44 ee 9b ff ff  ...........D....
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6028: 00 00 00 00 00 00 00 00 c0 c4 93 44 ee 9b ff ff  ...........D....
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6038: 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  @...............
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6048: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:15:36 zfstest kernel: Object   ffff9bec159d6058: c0 c7 8c c0 ff ff ff ff 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:36 zfstest kernel: Redzone  ffff9bec159d6068: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:36 zfstest kernel: Padding  ffff9bec159d6078: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:36 zfstest kernel: CPU: 91 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:36 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:36 zfstest kernel: Call Trace:
Oct 18 20:15:36 zfstest kernel:  <TASK>
Oct 18 20:15:36 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:36 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:36 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:36 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:36 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:15:36 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:15:36 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:36 zfstest kernel: ? zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:15:36 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:15:36 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:15:36 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:15:36 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:15:36 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:36 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:15:36 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:15:36 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:15:36 zfstest kernel: do_shrink_slab (mm/shrinker.c:436)
Oct 18 20:15:36 zfstest kernel: shrink_slab (mm/shrinker.c:663)
Oct 18 20:15:36 zfstest kernel: shrink_node (mm/vmscan.c:5872 mm/vmscan.c:5901)
Oct 18 20:15:36 zfstest kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Oct 18 20:15:36 zfstest kernel: balance_pgdat (mm/vmscan.c:6707 mm/vmscan.c:6890)
Oct 18 20:15:36 zfstest kernel: ? finish_task_switch (./arch/x86/include/asm/paravirt.h:698 kernel/sched/sched.h:1419 kernel/sched/core.c:5158 kernel/sched/core.c:5276)
Oct 18 20:15:36 zfstest kernel: ? __x64_sys_sched_getscheduler (kernel/sched/core.c:5859)
Oct 18 20:15:36 zfstest kernel: ? _raw_spin_lock_irqsave (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
Oct 18 20:15:36 zfstest kernel: ? lock_timer_base (kernel/time/timer.c:1051)
Oct 18 20:15:36 zfstest kernel: kswapd (mm/vmscan.c:7159)
Oct 18 20:15:36 zfstest kernel: ? prepare_to_wait_event (kernel/sched/wait.c:383)
Oct 18 20:15:36 zfstest kernel: ? balance_pgdat (mm/vmscan.c:7093)
Oct 18 20:15:36 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:36 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:36 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:36 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:36 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:36 zfstest kernel:  </TASK>
Oct 18 20:15:36 zfstest kernel: FIX kmalloc-96: Restoring Left Redzone 0xffff9bec159d6000-0xffff9bec159d6007=0xcc
Oct 18 20:15:36 zfstest kernel: FIX kmalloc-96: Object at 0xffff9bec159d6008 not freed
Oct 18 20:15:37 zfstest kernel: =============================================================================
Oct 18 20:15:37 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:37 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:37 zfstest kernel: 0xffff9bec110d0000-0xffff9bec110d000f @offset=0. First byte 0x71 instead of 0xcc
Oct 18 20:15:37 zfstest kernel: Slab 0xffffe87080443400 objects=42 used=21 fp=0xffff9bec110d1c00 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:37 zfstest kernel: Object 0xffff9bec110d0040 @offset=64 fp=0xffff9bec110d1840
Oct 18 20:15:37 zfstest kernel: Redzone  ffff9bec110d0000: 71 07 cf 9d e8 c5 26 64 02 6b ad 05 28 b0 c7 82  q.....&d.k..(...
Oct 18 20:15:37 zfstest kernel: Redzone  ffff9bec110d0010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:37 zfstest kernel: Redzone  ffff9bec110d0020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:37 zfstest kernel: Redzone  ffff9bec110d0030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:37 zfstest kernel: Object   ffff9bec110d0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:37 zfstest kernel: Object   ffff9bec110d0050: 00 00 00 00 00 00 00 00 50 73 6d c0 ff ff ff ff  ........Psm.....
Oct 18 20:15:37 zfstest kernel: Object   ffff9bec110d0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:37 zfstest kernel: Object   ffff9bec110d0070: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:37 zfstest kernel: Redzone  ffff9bec110d0080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:37 zfstest kernel: Padding  ffff9bec110d0090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:37 zfstest kernel: Padding  ffff9bec110d00a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:37 zfstest kernel: Padding  ffff9bec110d00b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:37 zfstest kernel: CPU: 81 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:37 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:37 zfstest kernel: Call Trace:
Oct 18 20:15:37 zfstest kernel:  <TASK>
Oct 18 20:15:37 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:37 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:37 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:37 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:37 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:15:37 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:15:37 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:37 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:15:37 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:15:37 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:15:37 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:15:37 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:37 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:15:37 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:15:37 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:15:37 zfstest kernel: do_shrink_slab (mm/shrinker.c:436)
Oct 18 20:15:37 zfstest kernel: shrink_slab (mm/shrinker.c:663)
Oct 18 20:15:37 zfstest kernel: shrink_node (mm/vmscan.c:5872 mm/vmscan.c:5901)
Oct 18 20:15:37 zfstest kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Oct 18 20:15:37 zfstest kernel: balance_pgdat (mm/vmscan.c:6707 mm/vmscan.c:6890)
Oct 18 20:15:37 zfstest kernel: ? finish_task_switch (./arch/x86/include/asm/paravirt.h:698 kernel/sched/sched.h:1419 kernel/sched/core.c:5158 kernel/sched/core.c:5276)
Oct 18 20:15:37 zfstest kernel: ? __x64_sys_sched_getscheduler (kernel/sched/core.c:5859)
Oct 18 20:15:37 zfstest kernel: ? _raw_spin_lock_irqsave (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
Oct 18 20:15:37 zfstest kernel: ? lock_timer_base (kernel/time/timer.c:1051)
Oct 18 20:15:37 zfstest kernel: kswapd (mm/vmscan.c:7159)
Oct 18 20:15:37 zfstest kernel: ? prepare_to_wait_event (kernel/sched/wait.c:383)
Oct 18 20:15:37 zfstest kernel: ? balance_pgdat (mm/vmscan.c:7093)
Oct 18 20:15:37 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:37 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:37 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:37 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:37 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:37 zfstest kernel:  </TASK>
Oct 18 20:15:37 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bec110d0000-0xffff9bec110d000f=0xcc
Oct 18 20:15:37 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bec110d0040 not freed
Oct 18 20:15:38 zfstest kernel: =============================================================================
Oct 18 20:15:38 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:38 zfstest kernel: 0xffff9bec6168a000-0xffff9bec6168a00f @offset=0. First byte 0xe3 instead of 0xcc
Oct 18 20:15:38 zfstest kernel: Slab 0xffffe8708185a280 objects=42 used=23 fp=0xffff9bec6168ab80 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:38 zfstest kernel: Object 0xffff9bec6168a040 @offset=64 fp=0xffff9bec6168bf00
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6168a000: e3 41 57 24 d8 38 c2 16 a6 6d f7 5d 4f 8d 5b bd  .AW$.8...m.]O.[.
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6168a010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6168a020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6168a030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6168a040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6168a050: 00 00 00 00 00 00 00 00 50 73 6d c0 ff ff ff ff  ........Psm.....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6168a060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6168a070: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6168a080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec6168a090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec6168a0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec6168a0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: CPU: 43 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:38 zfstest kernel: Call Trace:
Oct 18 20:15:38 zfstest kernel:  <TASK>
Oct 18 20:15:38 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:38 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:38 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:38 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:38 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:15:38 zfstest kernel: ? zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:15:38 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:38 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:15:38 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:15:38 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:15:38 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:15:38 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:38 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:15:38 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:15:38 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:15:38 zfstest kernel: do_shrink_slab (mm/shrinker.c:436)
Oct 18 20:15:38 zfstest kernel: shrink_slab (mm/shrinker.c:663)
Oct 18 20:15:38 zfstest kernel: shrink_node (mm/vmscan.c:5872 mm/vmscan.c:5901)
Oct 18 20:15:38 zfstest kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Oct 18 20:15:38 zfstest kernel: balance_pgdat (mm/vmscan.c:6707 mm/vmscan.c:6890)
Oct 18 20:15:38 zfstest kernel: ? finish_task_switch (./arch/x86/include/asm/paravirt.h:698 kernel/sched/sched.h:1419 kernel/sched/core.c:5158 kernel/sched/core.c:5276)
Oct 18 20:15:38 zfstest kernel: ? __x64_sys_sched_getscheduler (kernel/sched/core.c:5859)
Oct 18 20:15:38 zfstest kernel: ? _raw_spin_lock_irqsave (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
Oct 18 20:15:38 zfstest kernel: ? lock_timer_base (kernel/time/timer.c:1051)
Oct 18 20:15:38 zfstest kernel: kswapd (mm/vmscan.c:7159)
Oct 18 20:15:38 zfstest kernel: ? prepare_to_wait_event (kernel/sched/wait.c:383)
Oct 18 20:15:38 zfstest kernel: ? balance_pgdat (mm/vmscan.c:7093)
Oct 18 20:15:38 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:38 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:38 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:38 zfstest kernel:  </TASK>
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bec6168a000-0xffff9bec6168a00f=0xcc
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bec6168a040 not freed
Oct 18 20:15:38 zfstest kernel: =============================================================================
Oct 18 20:15:38 zfstest kernel: BUG kmalloc-32 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:38 zfstest kernel: 0xffff9bec0f041000-0xffff9bec0f04100f @offset=0. First byte 0x4 instead of 0xcc
Oct 18 20:15:38 zfstest kernel: Slab 0xffffe870803c1040 objects=42 used=18 fp=0xffff9bec0f041920 flags=0xfffffc0000200(workingset|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:38 zfstest kernel: Object 0xffff9bec0f041020 @offset=32 fp=0xffff9bec0f041860
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec0f041000: 04 d3 b2 0a eb 4b 06 1c 15 d9 ab ac 48 54 18 e6  .....K......HT..
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec0f041010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec0f041020: 00 00 00 00 01 00 00 00 40 72 5d 49 ee 9b ff ff  ........@r]I....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec0f041030: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec0f041040: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec0f041050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: CPU: 43 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:38 zfstest kernel: Call Trace:
Oct 18 20:15:38 zfstest kernel:  <TASK>
Oct 18 20:15:38 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:38 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:38 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:38 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:38 zfstest kernel: ? nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:38 zfstest kernel: ? nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:38 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:38 zfstest kernel: ? _raw_spin_unlock_irqrestore (./include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:194)
Oct 18 20:15:38 zfstest kernel: nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:38 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:15:38 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:15:38 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:15:38 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:15:38 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:38 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:15:38 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:15:38 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:15:38 zfstest kernel: do_shrink_slab (mm/shrinker.c:436)
Oct 18 20:15:38 zfstest kernel: shrink_slab (mm/shrinker.c:663)
Oct 18 20:15:38 zfstest kernel: shrink_node (mm/vmscan.c:5872 mm/vmscan.c:5901)
Oct 18 20:15:38 zfstest kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Oct 18 20:15:38 zfstest kernel: balance_pgdat (mm/vmscan.c:6707 mm/vmscan.c:6890)
Oct 18 20:15:38 zfstest kernel: ? finish_task_switch (./arch/x86/include/asm/paravirt.h:698 kernel/sched/sched.h:1419 kernel/sched/core.c:5158 kernel/sched/core.c:5276)
Oct 18 20:15:38 zfstest kernel: ? __x64_sys_sched_getscheduler (kernel/sched/core.c:5859)
Oct 18 20:15:38 zfstest kernel: ? _raw_spin_lock_irqsave (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
Oct 18 20:15:38 zfstest kernel: ? lock_timer_base (kernel/time/timer.c:1051)
Oct 18 20:15:38 zfstest kernel: kswapd (mm/vmscan.c:7159)
Oct 18 20:15:38 zfstest kernel: ? prepare_to_wait_event (kernel/sched/wait.c:383)
Oct 18 20:15:38 zfstest kernel: ? balance_pgdat (mm/vmscan.c:7093)
Oct 18 20:15:38 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:38 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:38 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:38 zfstest kernel:  </TASK>
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-32: Restoring Left Redzone 0xffff9bec0f041000-0xffff9bec0f04100f=0xcc
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-32: Object at 0xffff9bec0f041020 not freed
Oct 18 20:15:38 zfstest kernel: =============================================================================
Oct 18 20:15:38 zfstest kernel: BUG dmu_buf_impl_t (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:38 zfstest kernel: 0xffff9bec6ec04000-0xffff9bec6ec04007 @offset=0. First byte 0xd3 instead of 0xcc
Oct 18 20:15:38 zfstest kernel: Slab 0xffffe87081bb0100 objects=38 used=34 fp=0xffff9bec6ec05f80 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:38 zfstest kernel: Object 0xffff9bec6ec04008 @offset=8 fp=0xffff9bec6ec05c30
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6ec04000: d3 89 f5 03 7b 3a 03 dd                          ....{:..
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04008: 3f ff a1 3e aa 4c a2 9f ff ff ff ff ff ff ff ff  ?..>.L..........
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04018: 40 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  @...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04028: 00 d8 b0 54 ee 9b ff ff d8 8c 5a 03 ec 9b ff ff  ...T......Z.....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04038: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04048: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04058: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04068: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04078: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04088: 00 00 00 00 00 00 00 00 90 40 c0 6e ec 9b ff ff  [email protected]....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04098: 90 40 c0 6e ec 9b ff ff 00 00 00 00 00 00 00 00  [email protected]............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040a8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040b8: 00 00 00 00 00 00 00 00 c0 40 c0 6e ec 9b ff ff  [email protected]....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040c8: c0 40 c0 6e ec 9b ff ff 00 00 00 00 00 00 00 00  [email protected]............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040d8: 00 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040e8: 00 00 00 00 00 00 00 00 f5 45 65 34 00 00 00 00  .........Ee4....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec040f8: 00 00 00 00 00 00 00 00 00 41 c0 6e ec 9b ff ff  .........A.n....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04108: 00 41 c0 6e ec 9b ff ff 00 00 00 00 00 00 00 00  .A.n............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04118: 18 41 c0 6e ec 9b ff ff 18 41 c0 6e ec 9b ff ff  .A.n.....A.n....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04128: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04138: 00 00 00 00 00 00 00 00 98 01 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04148: 28 00 00 00 00 00 00 00 50 41 c0 6e ec 9b ff ff  (.......PA.n....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04158: 50 41 c0 6e ec 9b ff ff 00 01 00 00 00 00 ad de  PA.n............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04168: 22 01 00 00 00 00 ad de ff ff ff ff 00 00 00 00  "...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04178: 51 68 d5 2d 52 61 8b e0 00 00 00 00 00 00 00 00  Qh.-Ra..........
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec6ec04188: 01 00 00 00 00 00 00 00                          ........
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec6ec04190: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec6ec041a0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: CPU: 115 PID: 1498 Comm: dbu_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:38 zfstest kernel: Call Trace:
Oct 18 20:15:38 zfstest kernel:  <TASK>
Oct 18 20:15:38 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:38 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:38 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:38 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:38 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:38 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:38 zfstest kernel: kmem_cache_free (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4523)
Oct 18 20:15:38 zfstest kernel: spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:38 zfstest kernel: dbuf_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:3039) zfs
Oct 18 20:15:38 zfstest kernel: dnode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:676) zfs
Oct 18 20:15:38 zfstest kernel: dnode_buf_evict_async (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1351) zfs
Oct 18 20:15:38 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:15:38 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:15:38 zfstest kernel: ? dnode_create (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1947) zfs
Oct 18 20:15:38 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:15:38 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:38 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:38 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:38 zfstest kernel:  </TASK>
Oct 18 20:15:38 zfstest kernel: FIX dmu_buf_impl_t: Restoring Left Redzone 0xffff9bec6ec04000-0xffff9bec6ec04007=0xcc
Oct 18 20:15:38 zfstest kernel: FIX dmu_buf_impl_t: Object at 0xffff9bec6ec04008 not freed
Oct 18 20:15:38 zfstest kernel: =============================================================================
Oct 18 20:15:38 zfstest kernel: BUG dmu_buf_impl_t (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:38 zfstest kernel: 0xffff9bec07d00000-0xffff9bec07d00007 @offset=0. First byte 0x1c instead of 0xcc
Oct 18 20:15:38 zfstest kernel: Slab 0xffffe870801f4000 objects=38 used=31 fp=0xffff9bec07d00ba0 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:38 zfstest kernel: Object 0xffff9bec07d00008 @offset=8 fp=0xffff9bec07d00ef0
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec07d00000: 1c 3b 92 46 54 5e 38 d7                          .;.FT^8.
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00008: 63 13 5d 6f 71 ab 0c f9 ff ff ff ff ff ff ff ff  c.]oq...........
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00018: 40 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  @...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00028: 00 10 74 e6 ed 9b ff ff e8 ed 92 52 ee 9b ff ff  ..t........R....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00038: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00048: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00058: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00068: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00078: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00088: 00 00 00 00 00 00 00 00 90 00 d0 07 ec 9b ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00098: 90 00 d0 07 ec 9b ff ff 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000a8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000b8: 00 00 00 00 00 00 00 00 c0 00 d0 07 ec 9b ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000c8: c0 00 d0 07 ec 9b ff ff 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000d8: 00 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000e8: 00 00 00 00 00 00 00 00 f5 45 65 34 00 00 00 00  .........Ee4....
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d000f8: 00 00 00 00 00 00 00 00 00 01 d0 07 ec 9b ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00108: 00 01 d0 07 ec 9b ff ff 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00118: 18 01 d0 07 ec 9b ff ff 18 01 d0 07 ec 9b ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00128: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00138: 00 00 00 00 00 00 00 00 98 01 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00148: 28 00 00 00 00 00 00 00 50 01 d0 07 ec 9b ff ff  (.......P.......
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00158: 50 01 d0 07 ec 9b ff ff 00 01 00 00 00 00 ad de  P...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00168: 22 01 00 00 00 00 ad de ff ff ff ff 00 00 00 00  "...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00178: 97 b7 d3 64 f3 3f 8a 4e 00 00 00 00 00 00 00 00  ...d.?.N........
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec07d00188: 01 00 00 00 00 00 00 00                          ........
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec07d00190: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec07d001a0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: CPU: 115 PID: 1498 Comm: dbu_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:38 zfstest kernel: Call Trace:
Oct 18 20:15:38 zfstest kernel:  <TASK>
Oct 18 20:15:38 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:38 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:38 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:38 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:38 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:38 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:38 zfstest kernel: kmem_cache_free (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4523)
Oct 18 20:15:38 zfstest kernel: spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:38 zfstest kernel: dbuf_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dbuf.c:3039) zfs
Oct 18 20:15:38 zfstest kernel: dnode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:676) zfs
Oct 18 20:15:38 zfstest kernel: dnode_buf_evict_async (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1351) zfs
Oct 18 20:15:38 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:15:38 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:15:38 zfstest kernel: ? dnode_create (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1947) zfs
Oct 18 20:15:38 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:15:38 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:38 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:38 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:38 zfstest kernel:  </TASK>
Oct 18 20:15:38 zfstest kernel: FIX dmu_buf_impl_t: Restoring Left Redzone 0xffff9bec07d00000-0xffff9bec07d00007=0xcc
Oct 18 20:15:38 zfstest kernel: FIX dmu_buf_impl_t: Object at 0xffff9bec07d00008 not freed
Oct 18 20:15:38 zfstest kernel: =============================================================================
Oct 18 20:15:38 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:38 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:38 zfstest kernel: 0xffff9bec1db38000-0xffff9bec1db3800f @offset=0. First byte 0x54 instead of 0xcc
Oct 18 20:15:38 zfstest kernel: Slab 0xffffe8708076ce00 objects=42 used=29 fp=0xffff9bec1db38580 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:38 zfstest kernel: Object 0xffff9bec1db38040 @offset=64 fp=0xffff9bec1db38580
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec1db38000: 54 75 ea fd a0 d0 a6 7c 21 f2 fe 43 c3 43 1c a6  Tu.....|!..C.C..
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec1db38010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec1db38020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec1db38030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec1db38040: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec1db38050: 80 85 b3 1d ec 9b ff ff 80 85 b3 1d ec 9b ff ff  ................
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec1db38060: 30 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00  0...............
Oct 18 20:15:38 zfstest kernel: Object   ffff9bec1db38070: 03 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00  ................
Oct 18 20:15:38 zfstest kernel: Redzone  ffff9bec1db38080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec1db38090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec1db380a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: Padding  ffff9bec1db380b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:38 zfstest kernel: CPU: 99 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:38 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:38 zfstest kernel: Call Trace:
Oct 18 20:15:38 zfstest kernel:  <TASK>
Oct 18 20:15:38 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:38 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:38 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:38 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:38 zfstest kernel: ? zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:15:38 zfstest kernel: ? zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:15:38 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:38 zfstest kernel: zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:15:38 zfstest kernel: zfs_acl_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:506) zfs
Oct 18 20:15:38 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:15:38 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:15:38 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:15:38 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:15:38 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:38 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:15:38 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:15:38 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:15:38 zfstest kernel: do_shrink_slab (mm/shrinker.c:436)
Oct 18 20:15:38 zfstest kernel: shrink_slab (mm/shrinker.c:663)
Oct 18 20:15:38 zfstest kernel: shrink_node (mm/vmscan.c:5872 mm/vmscan.c:5901)
Oct 18 20:15:38 zfstest kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Oct 18 20:15:38 zfstest kernel: balance_pgdat (mm/vmscan.c:6707 mm/vmscan.c:6890)
Oct 18 20:15:38 zfstest kernel: ? finish_task_switch (./arch/x86/include/asm/paravirt.h:698 kernel/sched/sched.h:1419 kernel/sched/core.c:5158 kernel/sched/core.c:5276)
Oct 18 20:15:38 zfstest kernel: ? __x64_sys_sched_getscheduler (kernel/sched/core.c:5859)
Oct 18 20:15:38 zfstest kernel: ? _raw_spin_lock_irqsave (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
Oct 18 20:15:38 zfstest kernel: ? lock_timer_base (kernel/time/timer.c:1051)
Oct 18 20:15:38 zfstest kernel: kswapd (mm/vmscan.c:7159)
Oct 18 20:15:38 zfstest kernel: ? prepare_to_wait_event (kernel/sched/wait.c:383)
Oct 18 20:15:38 zfstest kernel: ? balance_pgdat (mm/vmscan.c:7093)
Oct 18 20:15:38 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:38 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:38 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:38 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:38 zfstest kernel:  </TASK>
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bec1db38000-0xffff9bec1db3800f=0xcc
Oct 18 20:15:38 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bec1db38040 not freed
Oct 18 20:15:39 zfstest kernel: =============================================================================
Oct 18 20:15:39 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:39 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:39 zfstest kernel: 0xffff9bec75fb6001-0xffff9bec75fb600f @offset=1. First byte 0xf8 instead of 0xcc
Oct 18 20:15:39 zfstest kernel: Slab 0xffffe87081d7ed80 objects=42 used=21 fp=0xffff9bec75fb6580 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:39 zfstest kernel: Object 0xffff9bec75fb6040 @offset=64 fp=0xffff9bec75fb6580
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec75fb6000: cc f8 7d 52 37 56 51 b3 06 2e cf ca 8e 96 ea b7  ..}R7VQ.........
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec75fb6010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec75fb6020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec75fb6030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec75fb6040: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec75fb6050: 80 65 fb 75 ec 9b ff ff 80 65 fb 75 ec 9b ff ff  .e.u.....e.u....
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec75fb6060: 30 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00  0...............
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec75fb6070: 03 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00  ................
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec75fb6080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:39 zfstest kernel: Padding  ffff9bec75fb6090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:39 zfstest kernel: Padding  ffff9bec75fb60a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:39 zfstest kernel: Padding  ffff9bec75fb60b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:39 zfstest kernel: CPU: 88 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:39 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:39 zfstest kernel: Call Trace:
Oct 18 20:15:39 zfstest kernel:  <TASK>
Oct 18 20:15:39 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:39 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:39 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:39 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:39 zfstest kernel: ? zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:15:39 zfstest kernel: ? zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:15:39 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:39 zfstest kernel: zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:15:39 zfstest kernel: zfs_acl_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:506) zfs
Oct 18 20:15:39 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:15:39 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:15:39 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:15:39 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:15:39 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:39 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:15:39 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:15:39 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:15:39 zfstest kernel: do_shrink_slab (mm/shrinker.c:436)
Oct 18 20:15:39 zfstest kernel: shrink_slab (mm/shrinker.c:663)
Oct 18 20:15:39 zfstest kernel: shrink_node (mm/vmscan.c:5872 mm/vmscan.c:5901)
Oct 18 20:15:39 zfstest kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Oct 18 20:15:39 zfstest kernel: balance_pgdat (mm/vmscan.c:6707 mm/vmscan.c:6890)
Oct 18 20:15:39 zfstest kernel: ? finish_task_switch (./arch/x86/include/asm/paravirt.h:698 kernel/sched/sched.h:1419 kernel/sched/core.c:5158 kernel/sched/core.c:5276)
Oct 18 20:15:39 zfstest kernel: ? __x64_sys_sched_getscheduler (kernel/sched/core.c:5859)
Oct 18 20:15:39 zfstest kernel: ? _raw_spin_lock_irqsave (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
Oct 18 20:15:39 zfstest kernel: ? lock_timer_base (kernel/time/timer.c:1051)
Oct 18 20:15:39 zfstest kernel: kswapd (mm/vmscan.c:7159)
Oct 18 20:15:39 zfstest kernel: ? prepare_to_wait_event (kernel/sched/wait.c:383)
Oct 18 20:15:39 zfstest kernel: ? balance_pgdat (mm/vmscan.c:7093)
Oct 18 20:15:39 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:39 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:39 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:39 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:39 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:39 zfstest kernel:  </TASK>
Oct 18 20:15:39 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bec75fb6001-0xffff9bec75fb600f=0xcc
Oct 18 20:15:39 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bec75fb6040 not freed
Oct 18 20:15:39 zfstest kernel: =============================================================================
Oct 18 20:15:39 zfstest kernel: BUG kmalloc-32 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:39 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:39 zfstest kernel: 0xffff9bec2a1de000-0xffff9bec2a1de00f @offset=0. First byte 0xa5 instead of 0xcc
Oct 18 20:15:39 zfstest kernel: Slab 0xffffe87080a87780 objects=42 used=15 fp=0xffff9bec2a1de0e0 flags=0xfffffc0000200(workingset|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:39 zfstest kernel: Object 0xffff9bec2a1de020 @offset=32 fp=0xffff9bec2a1ded40
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec2a1de000: a5 8d 9c 69 d4 e0 83 10 b4 2a 86 cd f2 21 0d 58  ...i.....*...!.X
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec2a1de010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec2a1de020: 00 00 00 00 01 00 00 00 40 6c 34 0d ec 9b ff ff  ........@l4.....
Oct 18 20:15:39 zfstest kernel: Object   ffff9bec2a1de030: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:39 zfstest kernel: Redzone  ffff9bec2a1de040: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:39 zfstest kernel: Padding  ffff9bec2a1de050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:39 zfstest kernel: CPU: 59 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:39 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:39 zfstest kernel: Call Trace:
Oct 18 20:15:39 zfstest kernel:  <TASK>
Oct 18 20:15:39 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:39 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:39 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:39 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:39 zfstest kernel: ? nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:39 zfstest kernel: ? nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:39 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:39 zfstest kernel: ? _raw_spin_unlock_irqrestore (./include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:194)
Oct 18 20:15:39 zfstest kernel: nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:39 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:15:39 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:15:39 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:15:39 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:15:39 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:39 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:15:39 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:15:39 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:15:39 zfstest kernel: do_shrink_slab (mm/shrinker.c:436)
Oct 18 20:15:39 zfstest kernel: shrink_slab (mm/shrinker.c:663)
Oct 18 20:15:39 zfstest kernel: shrink_node (mm/vmscan.c:5872 mm/vmscan.c:5901)
Oct 18 20:15:39 zfstest kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Oct 18 20:15:39 zfstest kernel: balance_pgdat (mm/vmscan.c:6707 mm/vmscan.c:6890)
Oct 18 20:15:39 zfstest kernel: ? finish_task_switch (./arch/x86/include/asm/paravirt.h:698 kernel/sched/sched.h:1419 kernel/sched/core.c:5158 kernel/sched/core.c:5276)
Oct 18 20:15:39 zfstest kernel: ? __x64_sys_sched_getscheduler (kernel/sched/core.c:5859)
Oct 18 20:15:39 zfstest kernel: ? _raw_spin_lock_irqsave (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
Oct 18 20:15:39 zfstest kernel: ? lock_timer_base (kernel/time/timer.c:1051)
Oct 18 20:15:39 zfstest kernel: kswapd (mm/vmscan.c:7159)
Oct 18 20:15:39 zfstest kernel: ? prepare_to_wait_event (kernel/sched/wait.c:383)
Oct 18 20:15:39 zfstest kernel: ? balance_pgdat (mm/vmscan.c:7093)
Oct 18 20:15:39 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:39 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:39 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:39 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:39 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:39 zfstest kernel:  </TASK>
Oct 18 20:15:39 zfstest kernel: FIX kmalloc-32: Restoring Left Redzone 0xffff9bec2a1de000-0xffff9bec2a1de00f=0xcc
Oct 18 20:15:39 zfstest kernel: FIX kmalloc-32: Object at 0xffff9bec2a1de020 not freed
Oct 18 20:15:40 zfstest kernel: =============================================================================
Oct 18 20:15:40 zfstest kernel: BUG kmalloc-32 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:40 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:40 zfstest kernel: 0xffff9bec42641000-0xffff9bec4264100f @offset=0. First byte 0x19 instead of 0xcc
Oct 18 20:15:40 zfstest kernel: Slab 0xffffe87081099040 objects=42 used=19 fp=0xffff9bec42641bc0 flags=0xfffffc0000200(workingset|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:40 zfstest kernel: Object 0xffff9bec42641020 @offset=32 fp=0xffff9bec426415c0
Oct 18 20:15:40 zfstest kernel: Redzone  ffff9bec42641000: 19 ea 29 0d 99 e6 c5 03 4f e2 76 72 82 c8 c1 69  ..).....O.vr...i
Oct 18 20:15:40 zfstest kernel: Redzone  ffff9bec42641010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec42641020: 00 00 00 00 01 00 00 00 00 0d 8e 03 ec 9b ff ff  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec42641030: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:40 zfstest kernel: Redzone  ffff9bec42641040: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:40 zfstest kernel: Padding  ffff9bec42641050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:40 zfstest kernel: CPU: 100 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:40 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:40 zfstest kernel: Call Trace:
Oct 18 20:15:40 zfstest kernel:  <TASK>
Oct 18 20:15:40 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:40 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:40 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:40 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:40 zfstest kernel: ? nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:40 zfstest kernel: ? nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:40 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:40 zfstest kernel: ? _raw_spin_unlock_irqrestore (./include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:194)
Oct 18 20:15:40 zfstest kernel: nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:40 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:15:40 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:15:40 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:15:40 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:15:40 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:40 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:15:40 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:15:40 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:15:40 zfstest kernel: do_shrink_slab (mm/shrinker.c:436)
Oct 18 20:15:40 zfstest kernel: shrink_slab (mm/shrinker.c:663)
Oct 18 20:15:40 zfstest kernel: shrink_node (mm/vmscan.c:5872 mm/vmscan.c:5901)
Oct 18 20:15:40 zfstest kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Oct 18 20:15:40 zfstest kernel: balance_pgdat (mm/vmscan.c:6707 mm/vmscan.c:6890)
Oct 18 20:15:40 zfstest kernel: ? finish_task_switch (./arch/x86/include/asm/paravirt.h:698 kernel/sched/sched.h:1419 kernel/sched/core.c:5158 kernel/sched/core.c:5276)
Oct 18 20:15:40 zfstest kernel: ? __x64_sys_sched_getscheduler (kernel/sched/core.c:5859)
Oct 18 20:15:40 zfstest kernel: ? _raw_spin_lock_irqsave (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
Oct 18 20:15:40 zfstest kernel: ? lock_timer_base (kernel/time/timer.c:1051)
Oct 18 20:15:40 zfstest kernel: kswapd (mm/vmscan.c:7159)
Oct 18 20:15:40 zfstest kernel: ? prepare_to_wait_event (kernel/sched/wait.c:383)
Oct 18 20:15:40 zfstest kernel: ? balance_pgdat (mm/vmscan.c:7093)
Oct 18 20:15:40 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:40 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:40 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:40 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:40 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:40 zfstest kernel:  </TASK>
Oct 18 20:15:40 zfstest kernel: FIX kmalloc-32: Restoring Left Redzone 0xffff9bec42641000-0xffff9bec4264100f=0xcc
Oct 18 20:15:40 zfstest kernel: FIX kmalloc-32: Object at 0xffff9bec42641020 not freed
Oct 18 20:15:40 zfstest kernel: =============================================================================
Oct 18 20:15:40 zfstest kernel: BUG dnode_t (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:40 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:40 zfstest kernel: 0xffff9bec70c88000-0xffff9bec70c88007 @offset=0. First byte 0xe7 instead of 0xcc
Oct 18 20:15:40 zfstest kernel: Slab 0xffffe87081c32200 objects=32 used=8 fp=0xffff9bec70c8df48 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:40 zfstest kernel: Object 0xffff9bec70c88008 @offset=8 fp=0xffff9bec70c887f8
Oct 18 20:15:40 zfstest kernel: Redzone  ffff9bec70c88000: e7 5e 95 6c 82 98 38 33                          .^.l..83
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88008: 92 8c b1 6f 74 7c 68 54 c1 33 2d 21 ed 9b ff ff  ...ot|hT.3-!....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88018: 00 00 00 00 00 00 00 00 20 80 c8 70 ec 9b ff ff  ........ ..p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88028: 20 80 c8 70 ec 9b ff ff 00 00 00 00 00 00 00 00   ..p............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88038: 00 01 00 00 00 00 ad de 22 01 00 00 00 00 ad de  ........".......
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88048: 01 d0 b8 33 ed 9b ff ff 63 9e 00 00 00 00 00 00  ...3....c.......
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88058: 78 a4 ee eb ed 9b ff ff 38 e2 21 4a ec 9b ff ff  x.......8.!J....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88068: 00 06 b9 60 ed 9b ff ff 13 00 00 00 b0 00 2c 01  ...`..........,.
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88078: 00 00 01 11 0a 00 02 00 00 04 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88088: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88098: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880a8: 00 6b 00 00 00 00 00 00 00 00 6b 6b 00 00 00 00  .k........kk....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880b8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880e8: 00 00 00 00 6b 6b 6b 6b 00 01 00 00 00 00 ad de  ....kkkk........
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c880f8: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88108: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88118: 22 01 00 00 00 00 ad de 00 01 00 00 00 00 ad de  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88128: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88138: 00 00 00 00 00 00 00 00 40 81 c8 70 ec 9b ff ff  [email protected]....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88148: 40 81 c8 70 ec 9b ff ff 00 00 00 00 6b 6b 6b 6b  @..p........kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88158: 00 00 00 00 00 00 00 00 98 01 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88168: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88178: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88188: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88198: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881a8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881b8: 22 01 00 00 00 00 ad de 98 01 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881c8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881d8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c881f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88208: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88218: 00 00 00 00 00 00 00 00 f5 45 65 34 6b 6b 6b 6b  .........Ee4kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88228: 00 00 00 00 6b 6b 6b 6b 30 82 c8 70 ec 9b ff ff  ....kkkk0..p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88238: 30 82 c8 70 ec 9b ff ff 00 00 00 00 6b 6b 6b 6b  0..p........kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88248: 48 82 c8 70 ec 9b ff ff 48 82 c8 70 ec 9b ff ff  H..p....H..p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88258: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88268: f5 45 65 34 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b  .Ee4kkkk....kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88278: 78 82 c8 70 ec 9b ff ff 78 82 c8 70 ec 9b ff ff  x..p....x..p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88288: 00 00 00 00 6b 6b 6b 6b 90 82 c8 70 ec 9b ff ff  ....kkkk...p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88298: 90 82 c8 70 ec 9b ff ff 00 00 00 00 00 00 00 00  ...p............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882a8: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b 6b  ............kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882b8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882c8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882d8: 00 00 00 00 00 00 00 00 e0 82 c8 70 ec 9b ff ff  ...........p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882e8: e0 82 c8 70 ec 9b ff ff 00 00 00 00 6b 6b 6b 6b  ...p........kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c882f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88308: 20 85 54 c0 ff ff ff ff 40 00 00 00 00 00 00 00   .T.....@.......
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88318: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88328: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88338: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88348: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88358: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88368: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88378: 00 00 00 00 6b 6b 6b 6b 00 00 00 00 00 00 00 00  ....kkkk........
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88388: 00 00 00 00 00 00 00 00 90 83 c8 70 ec 9b ff ff  ...........p....
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c88398: 90 83 c8 70 ec 9b ff ff 00 00 00 00 6b 6b 6b 6b  ...p........kkkk
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c883a8: 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c883b8: 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ad de  ................
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c883c8: 22 01 00 00 00 00 ad de 00 00 00 00 00 00 00 00  "...............
Oct 18 20:15:40 zfstest kernel: Object   ffff9bec70c883d8: 00 00 00 00 6b 6b 6b a5                          ....kkk.
Oct 18 20:15:40 zfstest kernel: Redzone  ffff9bec70c883e0: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:40 zfstest kernel: Padding  ffff9bec70c883f0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:40 zfstest kernel: CPU: 115 PID: 1498 Comm: dbu_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:40 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:40 zfstest kernel: Call Trace:
Oct 18 20:15:40 zfstest kernel:  <TASK>
Oct 18 20:15:40 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:40 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:40 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:40 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:40 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:40 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:40 zfstest kernel: kmem_cache_free (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4523)
Oct 18 20:15:40 zfstest kernel: spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:40 zfstest kernel: dnode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:648 (discriminator 1)) zfs
Oct 18 20:15:40 zfstest kernel: dnode_buf_evict_async (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1351) zfs
Oct 18 20:15:40 zfstest kernel: taskq_thread (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:994) spl
Oct 18 20:15:40 zfstest kernel: ? wake_up_state (kernel/sched/core.c:7084)
Oct 18 20:15:40 zfstest kernel: ? dnode_create (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/dnode.c:1947) zfs
Oct 18 20:15:40 zfstest kernel: ? task_done (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-taskq.c:526) spl
Oct 18 20:15:40 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:40 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:40 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:40 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:40 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:40 zfstest kernel:  </TASK>
Oct 18 20:15:40 zfstest kernel: FIX dnode_t: Restoring Left Redzone 0xffff9bec70c88000-0xffff9bec70c88007=0xcc
Oct 18 20:15:40 zfstest kernel: FIX dnode_t: Object at 0xffff9bec70c88008 not freed
Oct 18 20:15:47 zfstest kernel: =============================================================================
Oct 18 20:15:47 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:47 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:47 zfstest kernel: 0xffff9bedac286000-0xffff9bedac28600f @offset=0. First byte 0x10 instead of 0xcc
Oct 18 20:15:47 zfstest kernel: Slab 0xffffe87086b0a180 objects=42 used=34 fp=0xffff9bedac286580 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:15:47 zfstest kernel: Object 0xffff9bedac286040 @offset=64 fp=0x0000000000000000
Oct 18 20:15:47 zfstest kernel: Redzone  ffff9bedac286000: 10 4a 12 08 0a 98 5b 30 88 63 7a 79 75 11 48 15  .J....[0.czyu.H.
Oct 18 20:15:47 zfstest kernel: Redzone  ffff9bedac286010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:47 zfstest kernel: Redzone  ffff9bedac286020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:47 zfstest kernel: Redzone  ffff9bedac286030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:47 zfstest kernel: Object   ffff9bedac286040: 00 00 00 10 9f 01 1e 00 00 00 40 20 89 00 12 00  ..........@ ....
Oct 18 20:15:47 zfstest kernel: Object   ffff9bedac286050: 00 00 00 40 89 00 12 00 6b 6b 6b 6b 6b 6b 6b 6b  [email protected]
Oct 18 20:15:47 zfstest kernel: Object   ffff9bedac286060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:47 zfstest kernel: Object   ffff9bedac286070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:15:47 zfstest kernel: Redzone  ffff9bedac286080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:47 zfstest kernel: Padding  ffff9bedac286090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:47 zfstest kernel: Padding  ffff9bedac2860a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:47 zfstest kernel: Padding  ffff9bedac2860b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:47 zfstest kernel: CPU: 114 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:47 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:47 zfstest kernel: Call Trace:
Oct 18 20:15:47 zfstest kernel:  <TASK>
Oct 18 20:15:47 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:47 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:47 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:47 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:47 zfstest kernel: ? zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:15:47 zfstest kernel: ? zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:15:47 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:47 zfstest kernel: zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:15:47 zfstest kernel: zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:15:47 zfstest kernel: zfs_acl_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:506) zfs
Oct 18 20:15:47 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:15:47 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:15:47 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:15:47 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:15:47 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:47 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:15:47 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:15:47 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:15:47 zfstest kernel: do_shrink_slab (mm/shrinker.c:436)
Oct 18 20:15:47 zfstest kernel: shrink_slab (mm/shrinker.c:663)
Oct 18 20:15:47 zfstest kernel: shrink_node (mm/vmscan.c:5872 mm/vmscan.c:5901)
Oct 18 20:15:47 zfstest kernel: balance_pgdat (mm/vmscan.c:6707 mm/vmscan.c:6890)
Oct 18 20:15:47 zfstest kernel: ? finish_task_switch (./arch/x86/include/asm/paravirt.h:698 kernel/sched/sched.h:1419 kernel/sched/core.c:5158 kernel/sched/core.c:5276)
Oct 18 20:15:47 zfstest kernel: ? __x64_sys_sched_getscheduler (kernel/sched/core.c:5859)
Oct 18 20:15:47 zfstest kernel: ? _raw_spin_lock_irqsave (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
Oct 18 20:15:47 zfstest kernel: ? lock_timer_base (kernel/time/timer.c:1051)
Oct 18 20:15:47 zfstest kernel: kswapd (mm/vmscan.c:7159)
Oct 18 20:15:47 zfstest kernel: ? prepare_to_wait_event (kernel/sched/wait.c:383)
Oct 18 20:15:47 zfstest kernel: ? balance_pgdat (mm/vmscan.c:7093)
Oct 18 20:15:47 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:47 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:47 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:47 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:47 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:47 zfstest kernel:  </TASK>
Oct 18 20:15:47 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bedac286000-0xffff9bedac28600f=0xcc
Oct 18 20:15:47 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bedac286040 not freed
Oct 18 20:15:47 zfstest systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
Oct 18 20:15:48 zfstest kernel: =============================================================================
Oct 18 20:15:48 zfstest kernel: BUG lsm_inode_cache (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:48 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:48 zfstest kernel: 0xffff9bedfa59c000-0xffff9bedfa59c007 @offset=0. First byte 0x10 instead of 0xcc
Oct 18 20:15:48 zfstest kernel: Slab 0xffffe87087e96700 objects=42 used=35 fp=0xffff9bedfa59c548 flags=0x17ffffc0000200(workingset|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:15:48 zfstest kernel: Object 0xffff9bedfa59c008 @offset=8 fp=0xffff9bedfa59c8a8
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bedfa59c000: 10 d0 76 2a 55 2b 96 f8                          ..v*U+..
Oct 18 20:15:48 zfstest kernel: Object   ffff9bedfa59c008: 88 c1 27 43 ee 9b ff ff 00 00 00 00 00 00 00 00  ..'C............
Oct 18 20:15:48 zfstest kernel: Object   ffff9bedfa59c018: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:48 zfstest kernel: Object   ffff9bedfa59c028: 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:48 zfstest kernel: Object   ffff9bedfa59c038: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bedfa59c048: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:48 zfstest kernel: Padding  ffff9bedfa59c058: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:48 zfstest kernel: CPU: 47 PID: 304 Comm: ksoftirqd/47 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:48 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:48 zfstest kernel: Call Trace:
Oct 18 20:15:48 zfstest kernel:  <TASK>
Oct 18 20:15:48 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:48 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:48 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:48 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:48 zfstest kernel: ? rcu_core (kernel/rcu/tree.c:2542 kernel/rcu/tree.c:2809)
Oct 18 20:15:48 zfstest kernel: ? rcu_core (kernel/rcu/tree.c:2542 kernel/rcu/tree.c:2809)
Oct 18 20:15:48 zfstest kernel: kmem_cache_free (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4523)
Oct 18 20:15:48 zfstest kernel: ? rcu_core (./include/linux/bottom_half.h:33 kernel/rcu/tree.c:2554 kernel/rcu/tree.c:2809)
Oct 18 20:15:48 zfstest kernel: rcu_core (kernel/rcu/tree.c:2542 kernel/rcu/tree.c:2809)
Oct 18 20:15:48 zfstest kernel: ? try_to_wake_up (./include/linux/preempt.h:480 ./include/linux/preempt.h:480 kernel/sched/core.c:4229)
Oct 18 20:15:48 zfstest kernel: handle_softirqs (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:207 ./include/trace/events/irq.h:142 kernel/softirq.c:555)
Oct 18 20:15:48 zfstest kernel: run_ksoftirqd (./arch/x86/include/asm/paravirt.h:698 kernel/softirq.c:411 kernel/softirq.c:929 kernel/softirq.c:920)
Oct 18 20:15:48 zfstest kernel: smpboot_thread_fn (kernel/smpboot.c:164 (discriminator 3))
Oct 18 20:15:48 zfstest kernel: ? sort_range (kernel/smpboot.c:107)
Oct 18 20:15:48 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:48 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:48 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:48 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:48 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:48 zfstest kernel:  </TASK>
Oct 18 20:15:48 zfstest kernel: FIX lsm_inode_cache: Restoring Left Redzone 0xffff9bedfa59c000-0xffff9bedfa59c007=0xcc
Oct 18 20:15:48 zfstest kernel: FIX lsm_inode_cache: Object at 0xffff9bedfa59c008 not freed
Oct 18 20:15:48 zfstest kernel: =============================================================================
Oct 18 20:15:48 zfstest kernel: BUG kmalloc-32 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:48 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:48 zfstest kernel: 0xffff9bec6a515000-0xffff9bec6a51500f @offset=0. First byte 0x43 instead of 0xcc
Oct 18 20:15:48 zfstest kernel: Slab 0xffffe87081a94540 objects=42 used=5 fp=0xffff9bec6a515860 flags=0xfffffc0000200(workingset|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:48 zfstest kernel: Object 0xffff9bec6a515020 @offset=32 fp=0xffff9bec6a515260
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec6a515000: 43 11 1c b1 f1 1f 59 f5 59 e8 47 1c 93 06 95 df  C.....Y.Y.G.....
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec6a515010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec6a515020: 00 00 00 00 01 00 00 00 80 fa 64 30 ec 9b ff ff  ..........d0....
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec6a515030: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5  ........kkkkkkk.
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec6a515040: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:48 zfstest kernel: Padding  ffff9bec6a515050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:48 zfstest kernel: CPU: 34 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:48 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:48 zfstest kernel: Call Trace:
Oct 18 20:15:48 zfstest kernel:  <TASK>
Oct 18 20:15:48 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:48 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:48 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:48 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:48 zfstest kernel: ? nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:48 zfstest kernel: ? nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:48 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:48 zfstest kernel: ? _raw_spin_unlock_irqrestore (./include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:194)
Oct 18 20:15:48 zfstest kernel: nvlist_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/nvpair/nvpair.c:857) zfs
Oct 18 20:15:48 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:504) zfs
Oct 18 20:15:48 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:15:48 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:15:48 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:15:48 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:48 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:15:48 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:15:48 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:15:48 zfstest kernel: do_shrink_slab (mm/shrinker.c:436)
Oct 18 20:15:48 zfstest kernel: shrink_slab (mm/shrinker.c:663)
Oct 18 20:15:48 zfstest kernel: shrink_node (mm/vmscan.c:5872 mm/vmscan.c:5901)
Oct 18 20:15:48 zfstest kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Oct 18 20:15:48 zfstest kernel: balance_pgdat (mm/vmscan.c:6707 mm/vmscan.c:6890)
Oct 18 20:15:48 zfstest kernel: ? finish_task_switch (./arch/x86/include/asm/paravirt.h:698 kernel/sched/sched.h:1419 kernel/sched/core.c:5158 kernel/sched/core.c:5276)
Oct 18 20:15:48 zfstest kernel: ? __x64_sys_sched_getscheduler (kernel/sched/core.c:5859)
Oct 18 20:15:48 zfstest kernel: ? _raw_spin_lock_irqsave (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
Oct 18 20:15:48 zfstest kernel: ? lock_timer_base (kernel/time/timer.c:1051)
Oct 18 20:15:48 zfstest kernel: kswapd (mm/vmscan.c:7159)
Oct 18 20:15:48 zfstest kernel: ? prepare_to_wait_event (kernel/sched/wait.c:383)
Oct 18 20:15:48 zfstest kernel: ? balance_pgdat (mm/vmscan.c:7093)
Oct 18 20:15:48 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:48 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:48 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:48 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:48 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:48 zfstest kernel:  </TASK>
Oct 18 20:15:48 zfstest kernel: FIX kmalloc-32: Restoring Left Redzone 0xffff9bec6a515000-0xffff9bec6a51500f=0xcc
Oct 18 20:15:48 zfstest kernel: FIX kmalloc-32: Object at 0xffff9bec6a515020 not freed
Oct 18 20:15:48 zfstest systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
Oct 18 20:15:48 zfstest systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
Oct 18 20:15:48 zfstest kernel: =============================================================================
Oct 18 20:15:48 zfstest kernel: BUG kmalloc-64 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:48 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:48 zfstest kernel: 0xffff9bec4a922000-0xffff9bec4a92200f @offset=0. First byte 0xa instead of 0xcc
Oct 18 20:15:48 zfstest kernel: Slab 0xffffe870812a4880 objects=42 used=22 fp=0xffff9bec4a922d00 flags=0xfffffc0000240(workingset|head|node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:48 zfstest kernel: Object 0xffff9bec4a922040 @offset=64 fp=0xffff9bec4a922580
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec4a922000: 0a 98 12 9a be 27 4a db 3f e5 87 62 61 71 3a 86  .....'J.?..baq:.
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec4a922010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec4a922020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec4a922030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec4a922040: 00 00 00 10 9f 01 1e 00 00 00 40 20 89 00 12 00  ..........@ ....
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec4a922050: 00 00 00 40 89 00 12 00 6b 6b 6b 6b 6b 6b 6b 6b  [email protected]
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec4a922060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Oct 18 20:15:48 zfstest kernel: Object   ffff9bec4a922070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Oct 18 20:15:48 zfstest kernel: Redzone  ffff9bec4a922080: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:48 zfstest kernel: Padding  ffff9bec4a922090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:48 zfstest kernel: Padding  ffff9bec4a9220a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:48 zfstest kernel: Padding  ffff9bec4a9220b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:48 zfstest kernel: CPU: 34 PID: 945 Comm: kswapd0 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:48 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:48 zfstest kernel: Call Trace:
Oct 18 20:15:48 zfstest kernel:  <TASK>
Oct 18 20:15:48 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:48 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:48 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:48 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:48 zfstest kernel: ? zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:15:48 zfstest kernel: ? zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:15:48 zfstest kernel: kfree (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4569)
Oct 18 20:15:48 zfstest kernel: zfs_acl_node_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:488) zfs
Oct 18 20:15:48 zfstest kernel: zfs_acl_release_nodes (/usr/src/linux-headers-6.10.14-xxx/./include/linux/list.h:133) zfs
Oct 18 20:15:48 zfstest kernel: zfs_acl_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_acl.c:506) zfs
Oct 18 20:15:48 zfstest kernel: zfs_inode_destroy (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/zfs/zfs_znode.c:392) zfs
Oct 18 20:15:48 zfstest kernel: destroy_inode (fs/inode.c:312)
Oct 18 20:15:48 zfstest kernel: evict (fs/inode.c:722 (discriminator 2))
Oct 18 20:15:48 zfstest kernel: ? unlock_new_inode (fs/inode.c:850)
Oct 18 20:15:48 zfstest kernel: ? _raw_spin_unlock (./include/linux/spinlock_api_smp.h:143 kernel/locking/spinlock.c:186)
Oct 18 20:15:48 zfstest kernel: dispose_list (./include/linux/sched.h:1994 fs/inode.c:740)
Oct 18 20:15:48 zfstest kernel: prune_icache_sb (fs/inode.c:930)
Oct 18 20:15:48 zfstest kernel: super_cache_scan (fs/super.c:225)
Oct 18 20:15:48 zfstest kernel: do_shrink_slab (mm/shrinker.c:436)
Oct 18 20:15:48 zfstest kernel: shrink_slab (mm/shrinker.c:663)
Oct 18 20:15:48 zfstest kernel: shrink_node (mm/vmscan.c:5872 mm/vmscan.c:5901)
Oct 18 20:15:48 zfstest kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Oct 18 20:15:48 zfstest kernel: balance_pgdat (mm/vmscan.c:6707 mm/vmscan.c:6890)
Oct 18 20:15:48 zfstest kernel: ? finish_task_switch (./arch/x86/include/asm/paravirt.h:698 kernel/sched/sched.h:1419 kernel/sched/core.c:5158 kernel/sched/core.c:5276)
Oct 18 20:15:48 zfstest kernel: ? __x64_sys_sched_getscheduler (kernel/sched/core.c:5859)
Oct 18 20:15:48 zfstest kernel: ? _raw_spin_lock_irqsave (./arch/x86/include/asm/atomic.h:107 ./include/linux/atomic/atomic-arch-fallback.h:2170 ./include/linux/atomic/atomic-instrumented.h:1302 ./include/asm-generic/qspinlock.h:111 ./include/linux/spinlock.h:187 ./include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
Oct 18 20:15:48 zfstest kernel: ? lock_timer_base (kernel/time/timer.c:1051)
Oct 18 20:15:48 zfstest kernel: kswapd (mm/vmscan.c:7159)
Oct 18 20:15:48 zfstest kernel: ? prepare_to_wait_event (kernel/sched/wait.c:383)
Oct 18 20:15:48 zfstest kernel: ? balance_pgdat (mm/vmscan.c:7093)
Oct 18 20:15:48 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:48 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:48 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:48 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:48 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:48 zfstest kernel:  </TASK>
Oct 18 20:15:48 zfstest kernel: FIX kmalloc-64: Restoring Left Redzone 0xffff9bec4a922000-0xffff9bec4a92200f=0xcc
Oct 18 20:15:48 zfstest kernel: FIX kmalloc-64: Object at 0xffff9bec4a922040 not freed
Oct 18 20:15:50 zfstest kernel: =============================================================================
Oct 18 20:15:50 zfstest kernel: BUG zio_buf_comb_512 (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:50 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:50 zfstest kernel: 0xffff9bee08818000-0xffff9bee0881800f @offset=0. First byte 0x9 instead of 0xcc
Oct 18 20:15:50 zfstest kernel: Slab 0xffffe87088220600 objects=21 used=6 fp=0xffff9bee08818800 flags=0x17ffffc0000240(workingset|head|node=0|zone=2|lastcpupid=0x1fffff)
Oct 18 20:15:50 zfstest kernel: Object 0xffff9bee08818200 @offset=512 fp=0xffff9bee0881ca00
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818000: 09 fd 72 47 92 66 0d 2f 45 e4 ca b9 cc 5f 0a 50  ..rG.f./E...._.P
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818020: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818030: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818040: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818050: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818060: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818070: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818080: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818090: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088180f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818100: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818110: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818120: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818130: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818140: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818150: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818160: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818170: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818180: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818190: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181a0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181b0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181c0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181d0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181e0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee088181f0: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818200: 69 6d 70 6f 72 74 20 5f 70 6c 6f 74 6c 79 5f 75  import _plotly_u
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818210: 74 69 6c 73 2e 62 61 73 65 76 61 6c 69 64 61 74  tils.basevalidat
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818220: 6f 72 73 0a 0a 0a 63 6c 61 73 73 20 53 6f 75 72  ors...class Sour
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818230: 63 65 56 61 6c 69 64 61 74 6f 72 28 5f 70 6c 6f  ceValidator(_plo
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818240: 74 6c 79 5f 75 74 69 6c 73 2e 62 61 73 65 76 61  tly_utils.baseva
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818250: 6c 69 64 61 74 6f 72 73 2e 41 6e 79 56 61 6c 69  lidators.AnyVali
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818260: 64 61 74 6f 72 29 3a 0a 20 20 20 20 64 65 66 20  dator):.    def
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818270: 5f 5f 69 6e 69 74 5f 5f 28 73 65 6c 66 2c 20 70  __init__(self, p
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818280: 6c 6f 74 6c 79 5f 6e 61 6d 65 3d 22 73 6f 75 72  lotly_name="sour
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818290: 63 65 22 2c 20 70 61 72 65 6e 74 5f 6e 61 6d 65  ce", parent_name
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182a0: 3d 22 6c 61 79 6f 75 74 2e 6d 61 70 2e 6c 61 79  ="layout.map.lay
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182b0: 65 72 22 2c 20 2a 2a 6b 77 61 72 67 73 29 3a 0a  er", **kwargs):.
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182c0: 20 20 20 20 20 20 20 20 73 75 70 65 72 28 53 6f          super(So
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182d0: 75 72 63 65 56 61 6c 69 64 61 74 6f 72 2c 20 73  urceValidator, s
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182e0: 65 6c 66 29 2e 5f 5f 69 6e 69 74 5f 5f 28 0a 20  elf).__init__(.
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088182f0: 20 20 20 20 20 20 20 20 20 20 20 70 6c 6f 74 6c             plotl
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818300: 79 5f 6e 61 6d 65 3d 70 6c 6f 74 6c 79 5f 6e 61  y_name=plotly_na
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818310: 6d 65 2c 0a 20 20 20 20 20 20 20 20 20 20 20 20  me,.
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818320: 70 61 72 65 6e 74 5f 6e 61 6d 65 3d 70 61 72 65  parent_name=pare
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818330: 6e 74 5f 6e 61 6d 65 2c 0a 20 20 20 20 20 20 20  nt_name,.
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818340: 20 20 20 20 20 65 64 69 74 5f 74 79 70 65 3d 6b       edit_type=k
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818350: 77 61 72 67 73 2e 70 6f 70 28 22 65 64 69 74 5f  wargs.pop("edit_
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818360: 74 79 70 65 22 2c 20 22 70 6c 6f 74 22 29 2c 0a  type", "plot"),.
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818370: 20 20 20 20 20 20 20 20 20 20 20 20 2a 2a 6b 77              **kw
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818380: 61 72 67 73 2c 0a 20 20 20 20 20 20 20 20 29 0a  args,.        ).
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee08818390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Object   ffff9bee088183f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:50 zfstest kernel: Redzone  ffff9bee08818400: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818410: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818420: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818430: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818440: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818450: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818460: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818470: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818480: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818490: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088184f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818500: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818510: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818520: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818530: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818540: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818550: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818560: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818570: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818580: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee08818590: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: Padding  ffff9bee088185f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Oct 18 20:15:50 zfstest kernel: CPU: 66 PID: 1496 Comm: arc_evict Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:50 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:50 zfstest kernel: Call Trace:
Oct 18 20:15:50 zfstest kernel:  <TASK>
Oct 18 20:15:50 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:50 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:50 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:50 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:50 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:50 zfstest kernel: ? spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:50 zfstest kernel: kmem_cache_free (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4523)
Oct 18 20:15:50 zfstest kernel: spl_kmem_cache_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-kmem-cache.c:1382) spl
Oct 18 20:15:50 zfstest kernel: abd_free (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/abd.c:348) zfs
Oct 18 20:15:50 zfstest kernel: ? aggsum_add (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/aggsum.c:178) zfs
Oct 18 20:15:50 zfstest kernel: arc_hdr_free_abd (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:3217 (discriminator 2)) zfs
Oct 18 20:15:50 zfstest kernel: arc_evict_state (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:4275) zfs
Oct 18 20:15:50 zfstest kernel: arc_evict_cb (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/arc.c:6676 (discriminator 1)) zfs
Oct 18 20:15:50 zfstest kernel: zthr_procedure (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/zfs/zthr.c:246) zfs
Oct 18 20:15:50 zfstest kernel: ? zrl_is_locked (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/../zfs-2.2.6n/include/os/linux/spl/sys/mutex.h:54) zfs
Oct 18 20:15:50 zfstest kernel: ? spl_taskq_fini (/usr/src/linux-headers-6.10.14-xxx/./include/linux/err.h:50) spl
Oct 18 20:15:50 zfstest kernel: thread_generic_wrapper (/tmp/zfs-build--5aKIN1fq/BUILD/zfs-kmod-2.2.6n/_kmod_build_6.10.14-xxx/module/os/linux/spl/spl-thread.c:64) spl
Oct 18 20:15:50 zfstest kernel: kthread (kernel/kthread.c:389)
Oct 18 20:15:50 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:50 zfstest kernel: ret_from_fork (arch/x86/kernel/process.c:153)
Oct 18 20:15:50 zfstest kernel: ? kthread_park (kernel/kthread.c:342)
Oct 18 20:15:50 zfstest kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:254)
Oct 18 20:15:50 zfstest kernel:  </TASK>
Oct 18 20:15:50 zfstest kernel: FIX zio_buf_comb_512: Restoring Left Redzone 0xffff9bee08818000-0xffff9bee0881800f=0xcc
Oct 18 20:15:50 zfstest kernel: FIX zio_buf_comb_512: Object at 0xffff9bee08818200 not freed
Oct 18 20:15:54 zfstest kernel: =============================================================================
Oct 18 20:15:54 zfstest kernel: BUG lsm_inode_cache (Tainted: P    B      O      ): Left Redzone overwritten
Oct 18 20:15:54 zfstest kernel: -----------------------------------------------------------------------------
Oct 18 20:15:54 zfstest kernel: 0xffff9bec0398f000-0xffff9bec0398f007 @offset=0. First byte 0x84 instead of 0xcc
Oct 18 20:15:54 zfstest kernel: Slab 0xffffe870800e63c0 objects=42 used=42 fp=0x0000000000000000 flags=0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff)
Oct 18 20:15:54 zfstest kernel: Object 0xffff9bec0398f008 @offset=8 fp=0xffff9bec0398f428
Oct 18 20:15:54 zfstest kernel: Redzone  ffff9bec0398f000: 84 a0 bb 2b 7d 43 67 7e                          ...+}Cg~
Oct 18 20:15:54 zfstest kernel: Object   ffff9bec0398f008: 08 7f b2 33 ec 9b ff ff 00 00 00 00 00 00 00 00  ...3............
Oct 18 20:15:54 zfstest kernel: Object   ffff9bec0398f018: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:54 zfstest kernel: Object   ffff9bec0398f028: 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:54 zfstest kernel: Object   ffff9bec0398f038: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Oct 18 20:15:54 zfstest kernel: Redzone  ffff9bec0398f048: cc cc cc cc cc cc cc cc                          ........
Oct 18 20:15:54 zfstest kernel: Padding  ffff9bec0398f058: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Oct 18 20:15:54 zfstest kernel: CPU: 15 PID: 0 Comm: swapper/15 Tainted: P    B      O       6.10.14-xxx #1
Oct 18 20:15:54 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 20:15:54 zfstest kernel: Call Trace:
Oct 18 20:15:54 zfstest kernel:  <IRQ>
Oct 18 20:15:54 zfstest kernel: dump_stack_lvl (lib/dump_stack.c:117)
Oct 18 20:15:54 zfstest kernel: check_bytes_and_report (mm/slub.c:1196)
Oct 18 20:15:54 zfstest kernel: check_object (mm/slub.c:1307)
Oct 18 20:15:54 zfstest kernel: free_to_partial_list (mm/slub.c:1595 mm/slub.c:3281 mm/slub.c:4197)
Oct 18 20:15:54 zfstest kernel: ? rcu_core (kernel/rcu/tree.c:2542 kernel/rcu/tree.c:2809)
Oct 18 20:15:54 zfstest kernel: ? rcu_core (kernel/rcu/tree.c:2542 kernel/rcu/tree.c:2809)
Oct 18 20:15:54 zfstest kernel: kmem_cache_free (mm/slub.c:4401 mm/slub.c:4449 mm/slub.c:4523)
Oct 18 20:15:54 zfstest kernel: ? rcu_core (./include/linux/bottom_half.h:33 kernel/rcu/tree.c:2554 kernel/rcu/tree.c:2809)
Oct 18 20:15:54 zfstest kernel: rcu_core (kernel/rcu/tree.c:2542 kernel/rcu/tree.c:2809)
Oct 18 20:15:54 zfstest kernel: ? kvm_clock_read (./arch/x86/include/asm/preempt.h:94 arch/x86/kernel/kvmclock.c:80)
Oct 18 20:15:54 zfstest kernel: handle_softirqs (./arch/x86/include/asm/jump_label.h:27 ./include/linux/jump_label.h:207 ./include/trace/events/irq.h:142 kernel/softirq.c:555)
Oct 18 20:15:54 zfstest kernel: irq_exit_rcu (kernel/softirq.c:589 kernel/softirq.c:428 kernel/softirq.c:637 kernel/softirq.c:649)
Oct 18 20:15:54 zfstest kernel: sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1043 arch/x86/kernel/apic/apic.c:1043)
Oct 18 20:15:54 zfstest kernel:  </IRQ>
Oct 18 20:15:54 zfstest kernel:  <TASK>
Oct 18 20:15:54 zfstest kernel: asm_sysvec_apic_timer_interrupt (./arch/x86/include/asm/idtentry.h:702)
Oct 18 20:15:54 zfstest kernel: RIP: 0010:pv_native_safe_halt (arch/x86/kernel/paravirt.c:127)
Oct 18 20:15:54 zfstest kernel: Code: 09 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 22 d7 c3 66 66 2e 0f 1f 84 00 00 00 00 00 90 eb 07 0f 00 2d 0f fd 2c 00 fb f4 <c3> 0f 1f 40 00 83 ff 07 77 21 89 ff ff 24 fd f8 2e e1 b8 0f 21 f8
All code
========
   0: 09 c3                 or     %eax,%ebx
   2: 66 66 2e 0f 1f 84 00  data16 cs nopw 0x0(%rax,%rax,1)
   9: 00 00 00 00
   d: 66 90                 xchg   %ax,%ax
   f: 0f 22 d7              mov    %rdi,%cr2
  12: c3                    ret
  13: 66 66 2e 0f 1f 84 00  data16 cs nopw 0x0(%rax,%rax,1)
  1a: 00 00 00 00
  1e: 90                    nop
  1f: eb 07                 jmp    0x28
  21: 0f 00 2d 0f fd 2c 00  verw   0x2cfd0f(%rip)        # 0x2cfd37
  28: fb                    sti
  29: f4                    hlt
  2a:* c3                    ret  <-- trapping instruction
  2b: 0f 1f 40 00           nopl   0x0(%rax)
  2f: 83 ff 07              cmp    $0x7,%edi
  32: 77 21                 ja     0x55
  34: 89 ff                 mov    %edi,%edi
  36: ff 24 fd f8 2e e1 b8  jmp    *-0x471ed108(,%rdi,8)
  3d: 0f 21 f8              mov    %db7,%rax

Code starting with the faulting instruction
===========================================
   0: c3                    ret
   1: 0f 1f 40 00           nopl   0x0(%rax)
   5: 83 ff 07              cmp    $0x7,%edi
   8: 77 21                 ja     0x2b
   a: 89 ff                 mov    %edi,%edi
   c: ff 24 fd f8 2e e1 b8  jmp    *-0x471ed108(,%rdi,8)
  13: 0f 21 f8              mov    %db7,%rax
Oct 18 20:15:54 zfstest kernel: RSP: 0018:ffffbea8c0137ec0 EFLAGS: 00000202
Oct 18 20:15:54 zfstest kernel: RAX: 000000000000000f RBX: 000000000000000f RCX: 0000000000000000
Oct 18 20:15:54 zfstest kernel: RDX: 0000000000000000 RSI: ffffffffb9247b2c RDI: 0000000000371d74
Oct 18 20:15:54 zfstest kernel: RBP: ffff9bed00becd80 R08: 0000001881573345 R09: 0000000000000000
Oct 18 20:15:54 zfstest kernel: R10: 00000000000001c1 R11: 0000000000000007 R12: 0000000000000000
Oct 18 20:15:54 zfstest kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Oct 18 20:15:54 zfstest kernel: default_idle (./arch/x86/include/asm/paravirt.h:175 arch/x86/kernel/process.c:742)
Oct 18 20:15:54 zfstest kernel: default_idle_call (./include/linux/cpuidle.h:144 kernel/sched/idle.c:118)
Oct 18 20:15:54 zfstest kernel: do_idle (kernel/sched/idle.c:192 kernel/sched/idle.c:332)
Oct 18 20:15:54 zfstest kernel: cpu_startup_entry (kernel/sched/idle.c:429 (discriminator 1))
Oct 18 20:15:54 zfstest kernel: start_secondary (arch/x86/kernel/smpboot.c:202 arch/x86/kernel/smpboot.c:282)
Oct 18 20:15:54 zfstest kernel: common_startup_64 (arch/x86/kernel/head_64.S:421)
Oct 18 20:15:54 zfstest kernel:  </TASK>
Oct 18 20:15:54 zfstest kernel: FIX lsm_inode_cache: Restoring Left Redzone 0xffff9bec0398f000-0xffff9bec0398f007=0xcc
Oct 18 20:15:54 zfstest kernel: FIX lsm_inode_cache: Object at 0xffff9bec0398f008 not freed

Oct 19 '24 01:10 TheUbuntuGuy

I have a solution that works for me, would be great if others can validate: https://github.com/vpsfreecz/zfs/commit/74964ac17e0e5f461b74a0a847d87007d21c8d75

(I'm not 100% sure it will compile outside vpsfreecz kernel, but the problem is understood and I'll get to a PR for it once I hunt down some more issues, working on it furiously)

Oct 27 '24 00:10 snajpa

zfs zfs copied to clipboard

Kernel oops page fault triggered by Docker in arc_prune

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

zfs
zfs copied to clipboard