Kernel Panic: OpenZFS 2.1.0 Catalina

[sbytnar]

panic(cpu 4 caller 0xffffff802cb5a63c): "free_io_buf: bp(0xffffff8210c1c5f0) - bufstats.bufs_iobufinuse < 0"@/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.141.66/bsd/vfs/vfs_bio.c:4567 Backtrace (CPU 4), Frame : Return Address 0xffffff820deab030 : 0xffffff802c91b54d mach_kernel : _handle_debugger_trap + 0x49d 0xffffff820deab080 : 0xffffff802ca55d85 mach_kernel : _kdp_i386_trap + 0x155 0xffffff820deab0c0 : 0xffffff802ca4790e mach_kernel : _kernel_trap + 0x4ee 0xffffff820deab110 : 0xffffff802c8c1a40 mach_kernel : _return_from_trap + 0xe0 0xffffff820deab130 : 0xffffff802c91ac17 mach_kernel : _DebuggerTrapWithState + 0x17 0xffffff820deab230 : 0xffffff802c91b007 mach_kernel : _panic_trap_to_debugger + 0x227 0xffffff820deab280 : 0xffffff802d0c03bc mach_kernel : _panic + 0x54 0xffffff820deab2f0 : 0xffffff802cb5a63c mach_kernel : _free_io_buf + 0x16c 0xffffff820deab330 : 0xffffff7fb00a043c org.openzfsonosx.zfs : _ldi_vnode_io_intr + 0x2c 0xffffff820deab350 : 0xffffff7fb00a0403 org.openzfsonosx.zfs : _buf_strategy_vnode + 0x223 0xffffff820deab380 : 0xffffff7fb00e04b3 org.openzfsonosx.zfs : _vdev_disk_io_strategy + 0xe3 0xffffff820deab3c0 : 0xffffff7fb0153eca org.openzfsonosx.zfs : _zio_vdev_io_start + 0x1da 0xffffff820deab400 : 0xffffff7fb01500bc org.openzfsonosx.zfs : _zio_nowait + 0x15c 0xffffff820deab430 : 0xffffff7fb00eb854 org.openzfsonosx.zfs : _vdev_mirror_io_start + 0x124 0xffffff820deab480 : 0xffffff7fb0153eca org.openzfsonosx.zfs : _zio_vdev_io_start + 0x1da 0xffffff820deab4c0 : 0xffffff7fb01500bc org.openzfsonosx.zfs : _zio_nowait + 0x15c 0xffffff820deab4f0 : 0xffffff7fb003d6c6 org.openzfsonosx.zfs : _arc_read + 0xc36 0xffffff820deab5a0 : 0xffffff7fb004e59b org.openzfsonosx.zfs : _dbuf_read_impl + 0x27b 0xffffff820deab6b0 : 0xffffff7fb004e028 org.openzfsonosx.zfs : _dbuf_read + 0x338 0xffffff820deab720 : 0xffffff7fb0057134 org.openzfsonosx.zfs : _dmu_buf_hold + 0x44 0xffffff820deab760 : 0xffffff7fb0103365 org.openzfsonosx.zfs : _zap_lockdir + 0x35 0xffffff820deab7b0 : 0xffffff7fb0105207 org.openzfsonosx.zfs : _zap_cursor_retrieve + 0x157 0xffffff820deab840 : 0xffffff7fb0136b3e org.openzfsonosx.zfs : _zfs_readdir + 0x3be 0xffffff820deaba70 : 0xffffff7fb013ccc0 org.openzfsonosx.zfs : _zfs_vnop_readdir + 0xe0 0xffffff820deabaf0 : 0xffffff802cb95d19 mach_kernel : _vnode_readdir64 + 0x3e9 0xffffff820deabba0 : 0xffffff802cb56349 mach_kernel : _getattrlistbulk + 0xa59 0xffffff820deabf40 : 0xffffff802cf83517 mach_kernel : _unix_syscall64 + 0x287 0xffffff820deabfa0 : 0xffffff802c8c2206 mach_kernel : _hndl_unix_scall64 + 0x16 Kernel Extensions in backtrace: org.openzfsonosx.zfs(2.1)[D8D37025-5993-3A02-8E85-9537CACFBA27]@0xffffff7fb0037000->0xffffff7fb17dcfff dependency: com.apple.iokit.IOStorageFamily(2.1)[77F6B260-53DF-3DC0-B89F-5205E17FF986]@0xffffff7fad265000

This has been working great, forever, until now... Carbon Copy Cloner was backing up to a disk image.

The pool did not automount on reboot after this panic. zpool import worked. zpool status -v shows DEGRADED and permanent errors are in files associated with a disk image that is used to backup.

123GB free on the 3TB 5400RPM disk in question. smartctl status shows no problems. However, it will take several hours to get an extended smartctl test result.

[lundman]

Sounds like it tried to free a buf too many times, and the count of bugs went zero.

It is also curious you are in buf_strategy_vnode() so you have picked to use vnode style over iokit? The default is iokit - out of curiosity, any reason for this?

This should have been a one-time crash, you could try scrub to see if the files come back as OK