L1 MFU data should be accessed more than twice to be written to L2ARC

[tkittich]

Describe the feature would like to see added to OpenZFS

This is similar to #16499 but should be easier to implement. For l2arc_mfuonly = 2 (L2 cache all metadata (MRU+MFU) but only MFU data), L1 MFU data should be really frequently used (e.g. > 2) to be written to L2ARC. Otherwise less useful data could waste L2ARC write bandwidth and ruin potentially more useful data (e.g. metadata) in L2ARC.

How will this feature improve OpenZFS?

L2ARC hit rate should be way more than 50% with this feature and l2arc_mfuonly = 2. That is, most metadata and most frequently used data should eventually be in the L2ARC. With most metadata in L2ARC, the hit rate should already reach 50%.

Additional context

Perhaps l2arc_mfuonly = 10,20,30 could be used to specify how many times MFU data should be accessed before getting copied into L2ARC. But it would be best if this could be auto-tuned. For example, L2ARC should keep most metadata and store less and less accessed MFU data as L2ARC size gets bigger and bigger.