ZFS 2.0.7 release does not support LTS 5.15 kernels >= 5.15.37

[pittagurneyi]

System information

Type	Version/Name
Distribution Name	Gentoo
Distribution Version
Kernel Version	5.15.32-gentoo-r1
Architecture	x86_64
OpenZFS Version	zfs-2.0.7-r0-gentoo

Describe the problem you're observing

Although the release notes for 2.0.7 say that kernel versions up to 5.15 are supported, a function renaming in 5.15.37, which was backported from 5.16+, causes sendfile errors.

This has been reported for kernel versions 5.16+ in Issue 12971.

Also seen in Issue 13401 "iov_iter_fault_in_readable is now fault_in_iov_iter_readable".

Describe how to reproduce the problem

If I upgrade the kernel on any of my Gentoo systems to a newer 5.15 LTS release than 5.15.36, the system will be broken once I boot into it.

Emerging a package using portage will fail at the copy stage, i.e. when it moves the processed data from the sandbox into the ZFS root filesystem.

Installing/rebuilding/upgrading packages is no longer possible.

Include any warning/errors/backtraces from the system logs

See the linked issue.

Possible solution

After some searching I found a patch that solves the problem of the renamed function: "Linux 5.16 compat: Added mapping for iov_iter_fault_in_readable " by Rich Ercolani.

I applied it to 2.0.7 and the problem went away. Although I believe this patch could not cause undesirable side effects, I'd ask that a ZFS developer takes a look at it and would prefer a new official release to patching it myself.

I've been spending some time in the past months reading the issue tracker on problems occuring in ZFS 2.1.x and I'm not yet convinced I should upgrade from 2.0.7 to 2.1.x on my production machines. This leaves me in a bad spot. I don't want to risk 2.1.x yet, but bugs that affect 2.0.7 aren't getting fixed anymore.

At this point in time I'd ask that you, the ZFS developers, please reconsider creating another 2.0.x release for those of us who would like to stay on the LTS kernel 5.15 and ZFS 2.0.x for the foreseeable future. I'm guessing/hoping that I'm not the only one. For me stability and data safety are the most important factors.

Perhaps the release could be limited to bug fixes that are related to serious issues, like data corruption. Keeping it compatible with 5.15 would be great as well, as I intend to stay on 5.15 for at least a year. I would not want/need/require any patches that make kernels 5.16+ available.

There appear to be bug fixes that are not in any 2.0.x release yet, specifically those related to Issue 11900 "SEEK_DATA fails randomly":

• Issue 13329 "MySQL's ib_logfile1 cannot be created properly" fixed in 2.1.5 by Pull request 13338 "Corrected oversight in ZERO_RANGE behavior".
• Pull request 12746 "Default to zfs_dmu_offset_next_sync=1", included in 2.1.5

I believe they affect 2.0.x, but please correct me if I'm wrong.

If creating another release is not possible, then please adjust the release notes for 2.0.7, so the supported kernel versions are limited to <= 5.15.36.

Also, it would be great, if someone could look at the two linked pull requests and tell me if it would be safe for me to apply those (tests/... excluded) to 2.0.7 myself. They seem to be quite small.

In evaluating the current state of 2.1.x I'm facing the problem, that I don't always understand the severity of bug fixes, as this is unclear to me - most of the time - from the technical commit messages and I have to start digging into the issues that lead to that fix.

Therefore, the following is something I'd like to see as well, as it would help me understand the problems being fixed:

Issue 13612 "Meta: Include consequences of bugfixes in PR/commit messages"

Considering the severity of issues I have witnessed in the past, I would like to see a continously updated list of current and past bugs that affect data safety in any way. Categories that I'm most worried about are:

• silent data corruption
• data corruption

In my opinion I should be able to review the risks of using my currently selected version of ZFS by looking at a table. Perhaps one could put severity-related categories in column headings and list issues - time sorted - in column 1, while the affected ZFS versions are placed in the associated table cell. Issues that have no fixes yet should be highlighted.

Let's say such a table would look like this:

#Issue	silent data corruption	data corruption	kernel panic/system crash
12345, 2022-06-03	< 2.1.5, < 2.0.7	< 2.1.5, < 2.0.7
12344, 2022-05-02			< 2.0.6

Then, by looking at it, I'd immediately know to upgrade to 2.0.7 or 2.1.5 after reviewing the linked issue. If it were "only" kernel panics I could look at what kind of hardware, etc. is affected and then decide if I want to risk it or not.

However, if the table looked like this

#Issue	silent data corruption	data corruption	kernel panic/system crash
12345, 2022-06-03	< 2.1.5	< 2.1.5
12344, 2022-05-02	< 2.1.5, < 2.0.7	< 2.1.5, < 2.0.7
12343, 2022-04-01	unfixed, >= 2.1.3	unfixed, >= 2.1.3

then the decision I have to face becomes much harder, because in this theoretical scenario 2.0.7 hasn't received the fix for 12345, while 2.1.x still has unfixed serious issues. Neither staying at 2.0.7 nor moving to 2.1.5 are good options.

At the moment, this is how I feel when looking at 2.1.x.

If such metadata, in human readable format, could be included in commit messages, then autogenerating such a table would also be an option, correct?

Thank you all for your great work on this project!

[thesamesam]

As far as I'm aware, 2.0.x is functionally dead. I don't think it's wise to continue using it, even if it's not been officially declared EOL. This is why 2.1.x has been marked stable in Gentoo already, too.

I'm not the author of the patch there but I'm pretty confident it's safe. The dmu sync change is too. We could backport it in Gentoo I suppose.

What issues are you hitting in 2.1.x that are unfixed, exactly?

[gyakovlev]

FYI, 2.0.7 includes fixes for SEEK_DATA problem: https://github.com/openzfs/zfs/commit/a524f8d6afa141dd82bc60554610679614ea0f54 https://github.com/openzfs/zfs/commit/33fa3985a9791a6dcd2ee60ac9ea21486d068c0f

those commits are on 2.0.x branch and are included with 2.0.7 tag.

I agree that compat PR looks harmless to me, we can easily add it (in gentoo).

btw, I wanted to remove 2.0.7 from gentoo but haven't done so yet: check https://bugs.gentoo.org/850508

[pittagurneyi]

Thank you both for looking into it and answering some of my questions regarding the future of 2.0.7, specifically on Gentoo. However, as I have no problem keeping a local repository with the ebuild, I could stay longer on 2.0.7 if I wanted.

Perhaps I should explain my point of view better:

I don't need or want new features, not when they may introduce additional bugs and regressions. At least not on most of my systems.

When new features are added, it takes time to iron things out, which I consider normal. My question then is, what can be considered stable when it comes to a filesystem?

Data loss and corruption are two things I would consider unacceptable, no matter what circumstances. Therefore I don't care how old the base version of my current ZFS version is, I have no problem if it won't receive any more functional updates in the next months/years. I just want it to be perfect in one regard: Don't cause problems with my data.

That is basically what I'm asking here: Can there be a LTS version of ZFS that is supported for a longer time, which will only receive bug fixes that affect data integrity?

The only reason I'm on kernel 5.15 at the moment is because of fixes to mpt3sas and dmcrypt, which I was having problems with in some configurations. Some of my systems are and will stay on 5.10 for some time.

2.1.0 is a year old now. If I have the option, I would like to stay behind new feature-versions at least two years, when it comes to filesystems.

Not that I'm using it, because I distrusted such a new and complex feature from the beginning, but ZFS encryption tells me exactly what timeframe I should be looking at for new features to become stable. Last I checked someone mentioned in an issue, that encryption should be considered unstable and from what I read unusable, as massive data loss might happen. If it isn't stable yet, has it at least be deactivated and marked as experimental for users, so they may not accidentally activate it? From what I see, it has not.

If I read it right, then ZFS native encryption was merged into master on Aug 14, 2017 and released in 0.8.0 on May 23, 2019. So my math says I should stay behind new feature-releases at least 3+ years.

Perhaps create an option like NRPE has, here for activating new features that are not 5+ years old:

dont_blame_zfs

which has to be manually activated. That is a good hint that you are on your own from this point onward.

It is not only the feature itself but the integration into existing code, which might cause problems that are only discovered after some time.

Now, combine that with the hole_birth disaster from a few years back, which I believe is still deactivated in the code und was never fixed (?), then I think my question is not unreasonable.

Commit that deactivated it: Enable ignore_hole_birth module option by default with commit message:

Enable ignore_hole_birth by default until all known hole birth bugs have been resolved and relevant test cases added.

ZFS release that had that commit included: 0.7.0, released Jul 26, 2017.

The option was renamed afterwards, but here it is in master, still deactivated.

So now there are two options:

• either no one had time to look at it in years or
• developers don't consider all known hole birth bugs to be resolved.

I personally don't care that it is deactivated and am much happier knowing the developers take the time to be sure my data is safe rather than experimenting on it.

On the other side, how old is hole_birth as a feature? It was added in 0.6.4 on Apr 09, 2015. So 7+ years and not considered stable?

Am I risk-averse when it comes to my data? Yes. Can you really blame me?

[Evernow]

As far as I'm aware, 2.0.x is functionally dead. I don't think it's wise to continue using it, even if it's not been officially declared EOL.

I believe 2.0.x is end of life. According to this: https://github.com/openzfs/zfs/blob/master/RELEASES.md

2.0.x is not a LTS release, 2.1 is.

[pittagurneyi]

If that is correct, then practically, there is no LTS release.

Based on the definition provided in that file, 2.1.5 would both be the OpenZFS current AND OpenZFS LTS release, which makes no sense from my point of view.

A "LTS" release should provide you with a longer supported, proven and stable release, which receives only important patches. Such a release, to become proven and stable, would have had to mature in the past and now remain behind the current release. As 0.8 is no longer supported and 2.0 apparently EOL, that would leave only the current release, which, at the moment, can't really be a LTS release. It will/may become one in the future, if enough time has passed and if it doesn't still suffer from bugs.

If it were really stable, then it should not contain broken functionality, like ZFS encryption.

Consequently, if 2.0.x is EOL, then the filesystem ZFS has no stable release at the moment, at least not in the way that I would define stable when it comes to a filesystem.

I will not, out of principle, use an OpenZFS current release on production machines.

I consider OpenZFS current to be field testing of new code, which I consider unacceptable in most circumstances, when the data I'm working with is not irrelevant.

In that regard 0.8.x, 2.0.x and 2.1.x are all field-testing of new code when it comes to ZFS encryption. Only it is worse, because people know that it is not working perfectly and, for all intents and purposes, hide that fact from end users.

man 5 zpool-features on zfs 2.0.7 only has the following when it comes to encryption:

 encryption

     GUID                   com.datto:encryption
     READ-ONLY COMPATIBLE   no
     DEPENDENCIES           bookmark_v2, extensible_dataset

     This feature enables the creation and management of natively encrypted datasets.

     This feature becomes active when an encrypted dataset is created and will be returned
     to the enabled state when all datasets that use this feature are destroyed.

Where does it say that it is in an EXPERIMENTAL state?

If programs mangle your data, which might happen, then that is bad, but the possibility that the underlying layer, the filesystem, may have hidden severe bugs, which might silently and irrevocably corrupt your data, more often than not in an untraceable way, so you don't even know which data might have been affected and needs to be restored from backups, is not something I would ever consider inconsequential.

Considering the amount of changes that went into creating OpenZFS 2.0, I would have happily stayed at 0.8.x if it hadn't been EOL. Based on those changes and new features, I wouldn't have touched 2.0 until 2.2.x became two years old, but that wasn't really an option because of important bug fixes that never went into 0.8.x, forcing me to upgrade to 2.0.x. Now am I really forced to use a non-LTS version of ZFS again?

Could I please get an answer from those responsible for this project, exactly what kind of stability you intend to provide?

To be perfectly honest, I don't really see an alternative to ZFS for my use cases, but the levity with which user data has been treated in the past disturbs me greatly.

How long after discovering the first hole_birth bugs did it take for it to be turned off in the production release? I believe it was quite a long time. Now the mess with ZFS encryption is happening, which reminds me so much of the issues then.

I am truly grateful for ZFS and all the hard work that you, the developers, put into it, but I believe something has to change going forward.

[Evernow]

If that is correct, then practically, there is no LTS release.

Based on the definition provided in that file, 2.1.5 would both be the OpenZFS current AND OpenZFS LTS release, which makes no sense from my point of view.

A "LTS" release should provide you with a longer supported, proven and stable release, which receives only important patches. Such a release, to become proven and stable, would have had to mature in the past and now remain behind the current release. As 0.8 is no longer supported and 2.0 apparently EOL, that would leave only the current release, which, at the moment, can't really be a LTS release. It will/may become one in the future, if enough time has passed and if it doesn't still suffer from bugs.

Note I am just a user, but from memory what I gathered when this was being discussed is that right now OpenZFS LTS is the current release aswell, and when a future OpenZFS release comes around (3.0 I believe) then that will be the current release, and 2.1 will remain the LTS release until it is end of life on ~July 2023

This is similar to Ubuntu LTS right now, where the current and LTS release are right now the same. There are multiple projects that follow this same release formatting (RHEL, Firefox, Linux kernel, KDE Plasma, NVIDIA drivers, SUSE, Oracle Linux, Python, etc).

If it were really stable, then it should not contain broken functionality, like ZFS encryption.

Can you please link to the bug report? I run a RHEL (and Fedora) box with overall about 300TB of encrypted pools and have not experienced a single issue when using the latest ZFS releases.

I consider OpenZFS current to be field testing of new code, which I consider unacceptable in most circumstances, when the data I'm working with is not irrelevant.

It is not, look at the release notes: https://github.com/openzfs/zfs/releases

It is only receiving bug fixes on point releases, not new features. Unless you count new kernel support as new features that is, but it seems to be acceptable as long as the LTS release is also the current release.

Now am I really forced to use a non-LTS version of ZFS again?

OpenZFS 2.1 is supported till ~July 2023, after which you will have to update if you wish to keep receiving bug fixes.

[pittagurneyi]

@scineram and @Evernow: May I ask why you find my questions "negative"?

I believe we all depend on our data in different ways, so if for me even the most miniscule of unintended changes would be a big problem, is it truly unreasonable to ask these questions of a filesystem and its developers?

Note: Among other things I don't care to lose, I have scientific data on my systems and "just" want a good and stable filesystem without issues.

EDIT: Regarding the last comment:

I understand how LTS releases work. Normally however you have an "older" LTS release behind the current, in this case next LTS, release. That way you can choose to stay on the older one that didn't get new features "recently". At the moment there is no such "older" LTS release, which is why I was initially asking if 2.0.x could be supported a bit longer or what I am supposed to do.

OpenZFS current is where code lands that was tested in "master" only. That means mostly developers and automated testing environments. Then everyone that wants to use OpenZFS current can try it. There are typically bugs, which aren't caught in the initial testing. They get fixed in 2.1.x releases. That takes time.

I'd argue that the longer you can stay behind new features, the higher your chances are that you aren't affected by bugs, sometimes there are even fixed regressions due to earlier bug fixes, that only appear in later .x releases.

Just point me to an older LTS release which is still supported and I'm happy.

Regarding ZFS encryption:

Could you please do a bit of searching and for example read ZFS on Linux null pointer dereference #11679 ? In particular you'll find a comment by rincebrain:

My advice of not using native encryption hasn't changed in a long time, and probably won't for years after all the known issues are fixed, given the lack of work being put into it. This bug in particular was reported before it was even in a stable release, originally (https://github.com/openzfs/zfs/issues/8722), but here we are 3 years later.

My reproducer on Linux is on the recv side, but the workarounds discussed there are not specific to it. If you have a crash backtrace that isn't from the receive side, I'd love to see it, but I don't immediately see anyone having already reported that that I've forgotten.

In particular zfs send/recv is broken in some cases at least (if you don't backup your ZFS native encrypted data that way, you won't hit it ?!).

Then there are issues with the encryption keys, I believe they could be accidentally lost/overwritten/... ?

Also I think I read something about missing recovery tools. Pools, which were lost completely and had to be restored from backup. Things like that.

I believe the developers have a more comprehensive idea of what is and what isn't working.

I didn't ask them to fix it, I only questioned whether it is right not to warn users that these things might happen.

Typical "normal" users will probably not visit the github site of a filesystem that was installed with their Ubuntu distribution and search the issue tracker for hours and days. Instead I guess they'll silently assume everything is perfect, which to me is not an unreasonable assumption. Otherwise, if you had to constantly dig through everything your distribution consists of, how would you ever get things done?

I feel bad for those users, because when they try out new features, which sound exciting, like encryption, and get bitten by these bugs, your chances are high, that those users don't keep regular backups, if they do keep them at all.

As I said I don't use it, it is just an example of what I believe is questionable when it comes to a "stable" filesystem, as I believed I needed to explain/justify both my point of view and the questions I ask.

[rincebrain]

Re: hole_birth, my understanding is that all known issues with hole_birth were fixed at the same time as the tunable was introduced, and some vendors might have on their own products switched the default to allow using that metadata information again, but the fundamental problem with ameliorating hole_birth issues is that the data is incorrect on-disk for the lifetime of the data on that pool, it's just that it only matters when you do a send/recv and it decides based on that metadata that it doesn't need to send a hole along in the incremental, and you end up with data inconsistent on the recv side compared to the send side, silently. Oops. (I wrote a proof of concept that parsed zdb output to notice when this data was inconsistent, but it involved a combinatoric nightmare of overhead, so it was really just a proof that there's enough information to find it if you look, not something I'd suggest people run for fun.)

From that perspective, there's not really a good time to ever say "we don't need this any more" - I think you could, conceptually, add complex state tracking to the code that iterates over things for send, such that it will notice if it sees a hole with birth txg 0 when iterating over metadata from a txg > hole_birth activation and it wasn't there prior, but that's a lot of tracking to add for something which should, in theory, only become less common over time. (Of course, if people don't test that statement, more problems may have come in, but.)

I don't think I need to restate my opinions on how encryption is maintained here - I've said it ad nauseam in a lot of places at this point.

I will remark that I believe the disconnect is that most of the development work is focused around what the various employers' needs are, and most (all?) of them aren't, for example, using native encryption, from informally asking around, so the only time spent on it is volunteered, not paid, either by people who aren't paid to work on it or people who are but are also spending time above and beyond that...and there's precious little of that. And given that native encryption is the new dedup, I don't foresee most of them moving to use it anytime soon, either.

[pittagurneyi]

Thank you @rincebrain for explaining the hole_birth state of things and giving your opinion on the encryption feature.

I am aware, that the hole_birth problems manifested when sending sparse files via zfs send/recv. It forced me to use rsync for a while before hole_birth was deactivated in ZFS and I could revert to zfs send/recv. As I said before, I don't particularly care that it is deactivated, I'm quite happy that it doesn't have the chance to corrupt my backups anymore.

Regarding encryption: I read quite a lot of your and others comments over the last months, so I understand very well where the issues and the fact that they aren't getting fixed comes from.

The same reasons are probably responsible for my general problem with ZFS as well: New features are getting added and - in my opinion - not enough time is spent ironing out any and all existing problems. Of course new features are cool, they make for good marketing, etc. and my personal opinion is, that most companies don't really care that much about your (the end-non-company-users) data and tell themselves, it happens, not our problem.

This is the disconnect that I have severe problems with: People, like myself, that depend on their data, have very different priorities than the companies that fund parts of the development of ZFS. Does that however mean, that the project should only orient itself according to the wishes of those companies and not the non-company users?

I believe that can't be the case, especially as ZFS becomes more widely used and is integrated in end-user-oriented distributions like Ubuntu or systems like FreeNAS/TrueNAS.

There should be some sort of obligation to those users to keep their data safe. As a filesystem, it is exactly what - in my opinion - its main and only priority should always be.

While I could try reading through the issue tracker of the last year and find examples that aren't related to ZFS encryption, like parts of the SEEK_DATA problem, that are only now - please correct me if I'm wrong - getting fixed in 2.1.5, this is more of a general problem I think.

Is there a need for always keeping an old LTS version around, which predates the last massive feature and architectural changes to ZFS, so users have a stable and safe filesystem that reliably stores and protects their data? Should such a version always have any and all features, that are known to be buggy and cause data loss, disabled?

I believe I've made my point and don't want to repeat myself, so I simply vote yes and yes.

What do other people think?

Do you personally care about your data integrity and how much risk is acceptable to you?

Would you want to be warned about features that might cause the loss of parts or even all your data or would you prefer to remain ignorant, use it and hope for the best?

[ghost]

2.1 isn't "current" it's close to two years old. We're due for 2.2 IMO. New features from the past two years aren't in 2.1, just bug fixes.

[pittagurneyi]

I thought the release of 2.1.0 was on Jul 02, 2021, at least that is what the Releases page says. That would make 2.1.x a bit over a year old.

Personally I don't count the time when it is in (mostly) developers-only testing in master/rc-releases. Do you?

For me everyday usage in the real world is what counts when trying to get something to become very stable. I don't consider one year to be a long time when it comes to filesystems.

Issue #13329 is only 3 months ago, the fix had to be backported to 2.1.4, for example by Gentoo, and was only fixed in the last official release 21 days ago.

So there seem to be things that still get fixed. Some bugs might even be older and are not necessarily introduced in 2.1, but perhaps 2.0? It is hard to keep track as a non-developer, especially with sparse and technical commit notes.

Would the issue have happened in 0.8.x? I don't know.

I see things still getting fixed in the "OpenZFS current" releases and I stay away from them.

How am I to know which state which version of ZFS is in? The issue table I mentioned in my first post doesn't exist, right?

[ghost]

You're right, I was looking at zfs-2.0.0 not zfs-2.1.0 by mistake.

It's hard to keep track of issues in multiple branches even as a developer. For the most part I'm just periodically skimming through commits to master looking for things that look like a bug fix rather than a feature or performance improvement and seeing if they apply to the zfs-2.1-release branch. If so, I'll create a backport PR for it myself or mention it in a release staging PR if one is open. When we find an issue in a release branch it tends to still be applicable in master, so the issue gets fixed in master first and then backported eventually after it has been vetted in master for a while.

I don't bother trying to backport anything to 2.0 anymore because as far as I'm aware, there are no further releases planned on that branch. So much code has changed since that branched that the maintenance burden has become daunting and time consuming. You don't see bugs being fixed in 2.0 as much as in newer branches not because there aren't bugs, but because it's not being maintained.

2.0 was specifically intended to not be LTS because of the huge code restructuring and influx of features that took place for the 2.0 release. We anticipated there being wrinkles still to be ironed out.

2.1 has had time to mature the code base a bit further. It's the version that is used in FreeBSD 13 and TrueNAS CORE 13 and TrueNAS SCALE releases. It's the version that's actively maintained and used. It's the LTS branch. Bugs are being found and fixed in it because that's what people are using.

That said, I wouldn't be opposed if someone volunteered to be the maintainer of a 2.0 LTS branch.

[pittagurneyi]

Thank you for answering some of my questions.

Regarding 2.0 becoming LTS: I previously said, that I felt forced to upgrade to it, because 0.8 was EOL for a while already, severe bugs were found and fixes not backported. Still today I'd probably prefer 0.8 if it were maintained and switch to 2.1 after another year or so.

My problem is: There is no older LTS release I can stay on.

Regarding FreeBSD 13 using 2.1, okay I'll give you that, but that is still quite new and 13.1 was only released two months ago. I'll probably stay on 12.3 and then 12.4 for a bit before jumping to 13.2.

When it comes to TrueNAS however, I would not really consider them to be good judges of which ZFS version to use:

• They don't warn users about the problematic nature of ZFS encryption and were among the first I believe who integrated it into their software and promoted it (as replacement for GELI, which they called legacy encryption). Their users had quite a lot of fun with issues and data loss due to ZFS native encryption, from what I read in their forums and on reddit.
• They develop new ZFS features by themselves, that they integrate into their production releases before it has had a chance to hit the master branch in ZFS, so something I'd consider alpha-version code. I remembered that correctly, right? I believe there were quite a few issues opened by users that were affected by problems that originated from those features.

It appears I'm not the only one who thinks so, someone called mercenary_sysadmin wrote on ZFS self-corupts itself by using native encryption and snapshot replication:

TrueNAS likes to port in beta code, rather than sticking to actual production releases. This gets them in trouble every few years.

The history of actual production releases by the OpenZFS team is far better than the history of TrueNAS releases.

TrueNAS was just an example of where ZFS has found widespread usage - in hindsight, perhaps not a good example. But then I wasn't expecting someone to point to them as an example of production quality releases. Actually, now that I think about it, I stopped recommending it to friends 5-6 years ago, around the time when FreeNAS 10 (Corral) "happened".

EDIT: Yes I know where you work and that you will probably not be happy with my opinions on TrueNAS ... There isn't really anything I can do about it though.

[ghost]

As a TrueNAS developer I am personally responsible for ensuring that we have very few additional features on top of the upstream release. Most of them are backported from the upstream master branch after having been merged into OpenZFS. The only really big features are things we had to develop for SCALE to provide feature parity on Linux with long-existing functionality in FreeBSD, specifically the NFSv4 ACL support and consistent xattr namespacing. There are also several minor performance optimizations for our specific use cases that are not candidates for being backported to upstream stable releases. Other than that we occasionally include bug fixes from the release staging branch ahead of an upstream release. We do not have any huge extra features that we did not develop, such as encryption or draid, before they are in an upstream release. Any issues you have with ZFS in a release of TrueNAS are almost certainly issues you would have running an official release of OpenZFS.

[pittagurneyi]

Any issues you have with ZFS in a release of TrueNAS are almost certainly issues you would have running an official release of OpenZFS.

See FreeNAS (now TrueNAS) is no longer stable and PSA: silent data corruption in FreeNAS 12, fix incoming shortly in 12.0-U1.1.

The problematic code here was not developed by ixSystems, at least from what I read in the original pull request. However, the code therein has never been merged into ZFS master, not even to this day, and the original pull request was closed.

A new pull request is now open again and it still doesn't look like it will be merged in the near future.

TrueNAS 12.0-RELEASE was released on 2020-10-20 and the developers at ixSystems included alpha-version code, which caused silent data corruption for your users. Well, perhaps not you personally, depending on how long you've been working there, and even if so, it was surely not your decision to make at the time.

I still think there was stuff which the TrueNAS team developed themselves, which users reported problems to in this issue tracker and were sent away because ZFS didn't have that.

Perhaps nowadays TrueNAS has become reliable again and perhaps you contributed to that hypothetical fact, I guess I'll check it out again in a few years or so.

Now, I know I accidentally triggered this side-discussion, when I stupidly mentioned FreeNAS/TrueNAS while talking about ZFS stable releases, not anticipating that it would be used as an example of a stable environment that runs ZFS 2.1, but could we please leave this behind us now? I promise not to mention F/T/ixS again in this thread, unless someone tries to use it as an example w.r.t. stability.

[ghost]

Yep it wasn't my call to put it in, but I am the one who removed that when the guy who was working on it left. That was then, this is now. I absolutely share your sentiment about preferring to fix existing bugs over introducing shiny new features. I also think it is important to make sure that existing features work across platforms as much as possible, and are reasonably well tested and documented. You are welcome to help.

[gdevenyi]

Seems @pittagurneyi and I have very similar concerns about the purpose of versions, LTS and such, also storing the same kind of data. I can say that I'm still running 0.8.6 on all my storage servers due to the current state of the release branches.

A further complication to all the "what versions have what bugs" mess is that Ubuntu packages its own ZFS version, and doesn't change it, baking in bugs for LTS releases, I had a open issue about this for a long time https://github.com/openzfs/zfs/issues/10333

[stale[bot]]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[pittagurneyi]

So it seems that the future LTS release branch 2.1.x still contains unresolved data corruption bugs that are currently being worked on. I'm still running my own patched 2.0.7 on currently kernel 5.15.137 and, just to be safe, masked coreutils-9.x, replaced all hypervisors with Gentoo so I can choose the kernel, ZFS and coreutils version and migrated all FreeBSD systems to either Gentoo or VMs based on UFS. That was in July of 2022.

#15526

I tested the reproducer script and my setups don't seem to be affected in that configuration.

My notes in the mask file were:

# Coreutils 9.x exposes a ZFS (silent?) data corruption bug.
# There were fixes included in 2.0.7 and 2.1.2, but it is unclear
# if everything has been caught yet:
# 
## Issue
#
# "SEEK_DATA fails randomly"
# https://github.com/openzfs/zfs/issues/11900
#
# was (partially?) fixed by
#
# "Fix lseek(SEEK_DATA/SEEK_HOLE) mmap consistency"
# https://github.com/openzfs/zfs/pull/12724
# - got merged in 2.0.7 and 2.1.2
# - supersedes https://github.com/openzfs/zfs/pull/12710
#   - which wasn't needed anymore;
#   - this problem was related to mmap writes
#
## The following is a fix for a regression introduced by this fix (12724):
#
# "Restore dirty dnode detection logic"
# https://github.com/openzfs/zfs/pull/12745
# - got merged in 2.0.7 and 2.1.2
#
## This is a bug related to original issue 11900:
#
# "MySQL's ib_logfile1 cannot be created properly"
# https://github.com/openzfs/zfs/issues/13329
#
# which was fixed by
#
# "Corrected oversight in ZERO_RANGE behavior"
# https://github.com/openzfs/zfs/pull/13338
# - got merged in 2.1.5 BUT NOT 2.0.x !!!                   <-- UNFIXED in 2.0.x!
#
## This is a fix for a misunderstanding in the functionality of the ZFS code (bug)
## related to original issue 11900:
#
# "Default to zfs_dmu_offset_next_sync=1"
# https://github.com/openzfs/zfs/pull/12746
# - got merged in 2.1.5                                     <-- UNFIXED in 2.0.x!
#
## This is a bug related to original issue 11900 and probably fixed by 12746:
#
# "systemd test-copy test fails for lseek(SEEK_HOLE/SEEK_DATA) – punched holes not reported as such"
# https://github.com/openzfs/zfs/issues/13261
#
>=sys-apps/coreutils-9.0

It seems that it was indeed safer to not trust ZFS 2.1 and 2.2 yet and keep my data safe.

@gyakovlev: Going forward, could you please let me know what the intentions are regarding ZFS versions marked stable in Gentoo? I'm assuming this also depends on what ZFS does.

Could we now please open the discussion again on the necessity of a reliable, supported, older LTS release where those of us who depend on our data can "relax"? This is all giving me anxiety every time I open the ZFS issue tracker.

There seem to be possible data-corruption related fixes that should be backported to 2.0.7 (keeping it limited to kernel 5.15 is fine by me) until all issues in 2.1.x are ironed out.

Mentioned in #13755:

The most severe bug fixed in 2.1.6 is fixed by https://github.com/openzfs/zfs/commit/13f2b8fb92c23090b9f6e701c8471aef6b8e917b. I am not sure if it can cause corruption, but I am also not sure if it cannot cause corruption. There have been a few reports of pool corruption that might have been caused by it, but I have no proof beyond suspicion. However, it had been detected in issue reports from people running debug builds of ZFS, since it can trip an assertion. It causes undefined behavior that is difficult to model. This bug is present in all older versions with the btree code, which I think goes back to 0.8.0.

There was also another bug fixed by https://github.com/openzfs/zfs/commit/52afc3443d164101eea7fbded0d21c079a0d6a0e that also goes back to 0.8.0, which could cause free functions to be called on uninitialized data when processing encrypted data. However, that one is much more rare and requires a hash function fail to trigger it. There were no issue reports involving it. It would not surprise me if it has never been triggered anywhere in the world.

Lastly, we were calling free on stack memory in the skein code, which was fixed by https://github.com/openzfs/zfs/commit/a2163a96ae8708bb083e8da7658c02a7047516ba. That one was likely introduced in 0.7.0, but I did not check to be certain. The kernel free function probably harmlessly ignored it (since there were no issue reports), but I did not check to be certain.

Again, thanks to everyone who is spending their time on fixing the bugs that are found!

Also could somebody please answer me this: Is FreeBSD-14.0-RELEASE affected by any of these data corruption bugs?

[thesamesam]

I co-maintain ZFS in Gentoo with @gyakovlev. It's not really clear to me what versions of ZFS are LTS in reality. I don't really want to keep stable pinned to some version which doesn't even work with latest LTS kernels, as that will cause more harm than good as well.

If ZFS forms some sensible policy and keeps making releases for older ZFS branches, then we might be able to track that in stable, yeah (at the very least could probably keep the ebuilds around even if they were shadowed in stable by something newer).

We're really led by what ZFS decides to do and there's not much point in worrying about our side of things until that's straightened out.

just to be safe, masked coreutils-9.x,

Keep in mind that other software may use copy_file_range and friends.

Could we now please open the discussion again on the necessity of a reliable, supported, older LTS release where those of us who depend on our data can "relax"? This is all giving me anxiety every time I open the ZFS issue tracker.

I think this is a bigger issue regarding focus on bug fixes/QA vs new features and so on which I can't really offer input on.

Also, as discussed, the bad code may go back to 2011 or so anyway. I think you should reconsider using ZFS if you're not really able to rely on it at the moment. Trying to proxy/work around the fact you aren't able to trust it at the moment via using old versions and old kernels is a hack and not particularly helpful for anyone, I think.

[pittagurneyi]

Fair enough.

Keep in mind that other software may use copy_file_range and friends.

Absolutely correct and I've been waiting for a time when I felt like 2.1.13 was safe to use. Just hasn't happened yet.

I can't really use anything but ZFS at this point, because my entire setup including replication and offline backup is built around snapshots. The work it would take to migrate it all is just too much and everything else is working fine.

Don't get me wrong, I love ZFS and all its features, but the history of data corruption issues is not something I can just ignore.

There has to be a way for a ZFS version to exist, that is "trustworthy". It's not only me, @gdevenyi mentioned he is still on 0.8.6.

EDIT: rsync doesn't seem to use copy_file_range. Support for it seems to be work in progress: https://github.com/WayneD/rsync/issues/153 .

EDIT2:

Also, as discussed, the bad code may go back to 2011 or so anyway. I think you should reconsider using ZFS if you're not really able to rely on it at the moment. Trying to proxy/work around the fact you aren't able to trust it at the moment via using old versions and old kernels is a hack and not particularly helpful for anyone, I think.

It's not something I want to do, but something I currently see no real alternative to. Is my "hack" incomplete and not an ideal situation? Of course, there isn't any question about it. It is also not really maintainable and someday I'll again have to switch to a stable LTS release of ZFS that is maintained.

The question is just: Which version is it going to be and will there be solely bug fix focus on it? No optimizations, no features, nothing but the bare minimum to reduce the risk of data corruption so as not to accidentally introduce such bugs.