Is OpenZFS compatible with SPDK?

[recklessnl]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	11
Kernel Version	5.13.9
Architecture	x64
OpenZFS Version	2.1.1

Describe the problem you're observing

I'm currently trying to get NVMe-oF working using SPDK. My goal is to have four NVMe drives (all striped) managez by ZFS while utilizing SPDK. But I'm not sure if this is even possible, and an internet search come up empty. This is SPDK in its own words (link):

The Storage Performance Development Kit (SPDK) provides a set of tools and libraries for writing high performance, scalable, user-mode storage applications. It achieves high performance through the use of a number of key techniques: Moving all of the necessary drivers into userspace, which avoids syscalls and enables zero-copy access from the application. Polling hardware for completions instead of relying on interrupts, which lowers both total latency and latency variance. Avoiding all locks in the I/O path, instead relying on message passing. The bedrock of SPDK is a user space, polled-mode, asynchronous, lockless NVMe driver. This provides zero-copy, highly parallel access directly to an SSD from a user space application. The driver is written as a C library with a single public header. See NVMe Driver for more details. SPDK further provides a full block stack as a user space library that performs many of the same operations as a block stack in an operating system. This includes unifying the interface between disparate storage devices, queueing to handle conditions such as out of memory or I/O hangs, and logical volume management. See Block Device User Guide for more information. Finally, SPDK provides NVMe-oF, iSCSI, and vhost servers built on top of these components that are capable of serving disks over the network or to other processes. The standard Linux kernel initiators for NVMe-oF and iSCSI interoperate with these targets, as well as QEMU with vhost. These servers can be up to an order of magnitude more CPU efficient than other implementations. These targets can be used as examples of how to implement a high performance storage target, or used as the basis for production deployments.

But when I run the setup script, my NVMe drives stop working with ZFS and a whole system reboot (plus a reset of their startup script) is required to get it working again. What I have not tried yet is to enable SPDK first and then adding the drives to a zpool (as a cache first to check stability).

Is this supported / recommended at all? SPDK provides significant performance benefits over native Linux kernel when it comes to NVMe-oF.

[liyimeng]

@recklessnl Interesting use case. Do you add drives into zfs pools, and enable spdk?

[0xfd4d]

This happens because when spdk starting in default configuration, it unbinds nvme devices from in-kernel nvme driver to use it with vfio (spdk have it's own userspace nvme driver). As consequence, ZFS does not see any nvme drives. @recklessnl If you want to use spdk with zfs, you need to disable pci-e unbinding and configure spdk's aio or io_uring function to use zvol or file on dataset. spdk's io_uring on top of zvol works flawlessly on my system.

So, this is not ZFS issue.

[recklessnl]

@0xfd4d that is super interesting how you did that. Do you have a guide on how you set it up? That seems like you get the best of both worlds - ZFS management of devices and the speed of spdk (or are there massive speed differences in using native SPDK?)

[jacobloveless]

I second this request @0xfd4d ! Would love to see how you set that up!!!!