l2arc add option data to secondarycache or ignore caching of special metadata in l2arc

[kenthinson]

Describe the feature would like to see added to OpenZFS

Please note that I Have not been able to find a definitive answer about this in the docs or by searching issues on GitHub. If this is already properly handled then I request the docs be updated to address this information.

ZFS now has the special vdev. This can be setup to hold the metadata of the pool on flash disks speeding up the pool. However from what I have been reading on the ZFS docs if we add a l2arc to the pool it will attempt to cache the metadata as well.

So I propose that a data only secondarycache option be added. Example: secondarycache = all | none | metadata | data

OR

That the pool would be aware if the metadata came from a special device and not put it into l2arc if that is the case. This might be the more elegant behavior as it's transparent.

How will this feature improve OpenZFS?

My thinking is if the metadata is already stored on fast a flash special vdev. Then we don't need to cache it on l2arc. So having the option to only cache data on l2arc and not metadata could add a performance improvement. By reducing shuffling of metadata onto the l2arc drive and using more of the l2arc for data hits.

Additional context

I have run into an interesting possible performance problem. Where the l2arc might actually be slower to fetch metadata. As l2arc is a single large SSD. While my special vdev is 4x smaller SSDs. I am not sure how to test this though. At this point it is theoretical.

[gamanakis]

I wrote a patch that was merged a while ago, though not in a stable release yet. It introduced the "l2arc_exclude_special" zfs module property excluding data/metadata placed in a special vdev from being cached to L2ARC. https://github.com/openzfs/zfs/pull/12285/commits/39e038cdc61a9af2549fa3ed1f42c16dbcbe4ca6

[behlendorf]

@gamanakis I think this change would be reasonable to cherry pick in to the stable release. I've opened #13768 against the 2.1.6 staging branch.

[IvanVolosyuk]

I'm patching this change to 2.1.x ever since and it applies cleanly and works fine as far as I can see.