Enable zfs send|receive for encrypted root dataset

[barrkel]

Describe the feature would like to see added to OpenZFS

Enable zfs send | zfs receive of encrypted root datasets.

Currently, it is not possible to zfs send -wR | zfs receive -F an encrypted root dataset on each side.

To create the new filesystem with all properties, zfs send -w must be used, and since the root dataset already exists, -F must be used on the receiving side.

But zfs receive -F cannot replace an unencrypted filesystem with an encrypted one, nor delete an encrypted filesystem. It follows that no matter whether the target pool's root dataset is encrypted or unencrypted, it cannot receive an encrypted dataset.

How will this feature improve OpenZFS?

Enable transferring an encrypted root dataset, including all child filesystems that are inheriting their encryption from the root, from one vdev to another, such as when changing the pool size, composition or redundancy.

The only workaround I'm aware of is to reencrypt the data (i.e. don't use -w), and copy all the child filesystems, one by one, and manually update properties as necessary to match. You also lose all your snapshots.

[GenericRikka]

I believe this is feasible - zfs receive already handles raw encrypted streams, the limitation is that it refuses to replace a dataset with a different encryption state. Likely, the receive path just needs to allow this when -F is specified. I’ll investigate further and see if I can put together a patch.

[GenericRikka]

Update: prototype for receiving encrypted ↔ unencrypted with zfs send -w … | zfs receive -F …

I’ve been prototyping a gated way to let a full/RAW receive (-w) cross the encryption boundary on -F (blow-away) receives.

What I implemented so far

• Userspace opt-in flag: added a long option --allow-encryption-change that, when present, tags the receive with a boolean opt-in. I tried two plumbs:
  1. stashing a boolean in cmdprops/hidden args, and
  2. adding recv.allow_encryption_change=true to the rcvprops nvlist.
• Guard logic in userspace (libzfs_sendrecv.c::zfs_receive_one)
  • Keep the longstanding rule: incremental RAW onto an unencrypted fs is still rejected.
  • For full/newfs receives on an existing target with -F, allow crossing the encryption boundary only when:
    • the stream is RAW (-w), and
    • the opt-in is present.
  • (Earlier I also experimented with pre-flight checks of the destination pool’s feature@encryption and surfacing a clear hint to zpool upgrade. I backed that out temporarily to focus on the basic opt-in path.)
• Kernel side (first iteration, then pared back):
  • Initial approach plumbed a new drc_allow_enc_change through dmu_recv and relaxed checks in recv_begin_check_existing_impl() / dsl_dataset_clone_swap_check_impl(); also experimented with updating the directory’s encryption root/keystore mapping during clone-swap.
  • After feedback and compilation mismatches, I pared this back to just the userspace opt-in for now so I can validate the surface and the error paths before proposing kernel changes.

Current status / repro

• Repro (minimal):

zfs destroy -r src dst || true
zpool create -f src /tmp/src.img     # src can be encrypted
zpool create -f dst /tmp/dst.img     # dst is unencrypted
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase src/fs
echo ok > /src/fs/file
zfs snap src/fs@A

# Attempt to replace dst with raw encrypted stream
zfs send -w src/fs@A | zfs receive -F -u --allow-encryption-change dst/fs

• What I hit right now: an assert in userspace:

ASSERT at module/nvpair/fnvpair.c:140: fnvlist_add_boolean_value()
VERIFY0(nvlist_add_boolean_value(nvl, name, val)) failed (22)

This happens when the incoming stream has no properties nvlist, so rcvprops is NULL and my code unconditionally does:

fnvlist_add_boolean_value(rcvprops, "recv.allow_encryption_change", B_TRUE);

Fix is straightforward: ensure rcvprops = fnvlist_alloc() (or use cmdprops) before adding.

Open design questions for maintainers (ZFS Maintainers, @barrkel (issue opener), thoughts welcome)

1. Surface area: do you prefer a CLI long option, a zfs receive property (e.g., -o recv.allow_encryption_change=on), or an explicit ioctl bit? I can wire whichever you prefer.
2. Pool feature gating: should userspace
   • fail fast with a clear message if feature@encryption is disabled on the destination pool, or
   • try to auto-enable it (after an interactive confirmation or an explicit flag)? I’m leaning fail-fast + clear guidance.
3. Kernel involvement: are you comfortable keeping this mostly in userspace (i.e., relax the CLI guard for full/RAW + -F + opt-in) and letting the existing receive machinery do the rest, or do you want explicit kernel validation and/or a cookie bit (drc_allow_enc_change) so the kernel can make the policy decision?
4. Scope: my intent is to allow only full/RAW -F receives across the boundary (destroy+recreate), not incremental cross-boundary receives (those remain disallowed). Is that acceptable?
5. Root datasets: any caveats you want enforced for zpool/ROOT (boot environments), or is the general rule above sufficient when the target is unmounted or -u is used?

Next steps on my side

• Fix the nvlist assert (allocate before add) and resync the prototype.
• Re-add the minimal guard that allows -F boundary crossing only for RAW + opt-in, keeping incrementals blocked.
• If the approach is acceptable, I’ll put together zfstests covering:
  1. enc→unencrypted with -F + opt-in (succeeds),
  2. unencrypted→enc with -F + opt-in (succeeds),
  3. pool without feature@encryption (fails with guidance),
  4. incremental RAW onto unencrypted (still fails). Happy to adjust the plumbing based on your preferences (hidden arg vs. rcvprops vs. ioctl flag) and any security/UX concerns. Feedback appreciated!

[GenericRikka]

Status update (2025-09-30): pausing work; WIP patch attached for hand-off

I’m going to step back from this ticket for a bit while I focus on a time-critical FreeBSD/HPC modernization effort. I don’t want to go silent or block progress, so I’m uploading my current WIP patch and notes so anyone can pick this up — or so I can resume cleanly when capacity frees up.

What’s implemented so far

• Userspace opt-in: zfs receive --allow-encryption-change (plumbed via recvflags_t).
• Guarding in libzfs_sendrecv.c to allow -F full RAW (-w) receives to cross the encryption boundary only when the opt-in is present; incremental cross-boundary receives remain disallowed.
• Pool feature gate: friendly error if destination pool has feature@encryption=disabled.
• Kernel plumbing (WIP): pass an allow bit through receive path (dmu_recv_cookie.drc_allow_enc_change) and relax checks in recv_begin_check_existing_impl() / dsl_dataset_clone_swap_check_impl() for the force + raw + opt-in case. Also update encryption-root on clone-swap in the sync path (WIP).

Known issues / TODOs

• Recheck nvlist handling: ensure rcvprops/cmdprops are allocated before adding the opt-in (I hit an fnvlist_add_boolean_value assert earlier; addressed in the patch but worth double-checking).
• Kernel side needs careful review:
  • Verify mount/busy state and boot-env safety for root datasets (reject if mounted unless -u).
  • Audit encryption-root/keystore transitions on clone-swap.
  • Confirm no holes with embedded data, redaction, and resumable receives.
• Tests to add in zfstests:
  1. enc → unenc with -F + RAW + opt-in (success)
  2. unenc → enc with -F + RAW + opt-in (success)
  3. pool without feature@encryption (clear failure message)
  4. incremental RAW onto unencrypted (still fails)
  5. Root dataset cases with -u vs mounted target
• UX bikeshed: keep long option vs -o recv.allow_encryption_change=on vs ioctl bit—maintainer preference welcome.

WIP patch

allow-encryption-change-wip.patch I’ve attached the patch as allow-encryption-change-wip.patch. It includes userspace + kernel changes described above.

Repro I’ve been using (minimal)

zpool create -f src /tmp/src.img
zpool create -f dst /tmp/dst.img
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase src/fs
echo ok > /src/fs/file
zfs snap src/fs@A
zfs send -w src/fs@A | zfs receive -F -u --allow-encryption-change dst/fs

If someone wants to take this over now, please feel free to run with it. Otherwise I’ll circle back when my bandwidth returns. Thanks for the patience — and sorry for the pause after kicking this off.

—GenericRikka

[barrkel]

Thanks for your work @GenericRikka if I was in a position to push it forward I would.

[GenericRikka]

Appreciate it, thanks @barrkel! I’ve pushed the current state as a WIP and added a checklist so others can jump in. I’ll be heads-down on FreeBSD/HPC infra for a bit, but I’ll circle back when I have bandwidth in the not-too-distant future.

[arturpzol]

@GenericRikka

First of all - thank you for your work on this feature. It’s a really valuable addition, and I’m very interested in testing it.

While testing the allow-encryption-change-wip.patch, I encountered a similar issue to the one you mentioned earlier:

ASSERT at module/nvpair/fnvpair.c:140:fnvlist_add_boolean_value()
VERIFY0(nvlist_add_boolean_value(nvl, name, val)) failed (22)
  PID: 23756     COMM: zfs
  TID: 23756     NAME: zfs
Call trace:
/usr/lib/libzfs.so.6(libspl_backtrace+0x25)[0x7f56b736cdab]
/usr/lib/libzfs.so.6(libspl_assertf+0x175)[0x7f56b736cd5d]
/usr/lib/libnvpair.so.3(fnvlist_add_boolean_value+0x68)[0x7f56b6eee448]
/usr/lib/libzfs.so.6(+0x459a3)[0x7f56b734a9a3]
/usr/lib/libzfs.so.6(+0x48c79)[0x7f56b734dc79]
/usr/lib/libzfs.so.6(zfs_receive+0x102)[0x7f56b734ddc8]
zfs[0x41137b]
zfs[0x419c9e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f56b6724b45]
zfs[0x405bb9]
Aborted

It seems the patch might not be fully complete - specifically, rcvprops might not be allocated before calling fnvlist_add_boolean_value().

I added the following allocation for rcvprops:

+       /* If user opted in, add a boolean prop we can read in-kernel */
+       if (flags->allow_enc_change) {
+               if (rcvprops == NULL)
+                       rcvprops = fnvlist_alloc();
+               fnvlist_add_boolean_value(rcvprops, "recv.allow_encryption_change", B_TRUE);
+       }

After this change, the assert is gone, but I now get a segmentation fault:

zfs send -w src/fs@A | zfs receive -F -u --allow-encryption-change dst/fs
Segmentation fault

Would you mind checking whether the patch is complete or if there might be something else missing?

[GenericRikka]

Thanks for testing this and for the clear report, @arturpzol

You’re absolutely right on the issues you spotted:

• rcvprops needs to be allocated before fnvlist_add_boolean_value(); I’ll flip newprops accordingly as well.

• A bad merge left duplicate lzc_receive_with_heal(...) calls, which likely explains the segfault (second call consumes a half-read stream).

I’ll clean these up, re-test, and post an updated WIP patch. Timing note: I’m currently addressing review feedback on a separate FreeBSD/HPC upgrade series, so it may take me a bit — but this is on my radar and I’ll report back with a refreshed patch as soon as I can.

Thanks again for kicking the tires, your repro and notes are super helpful.

[GenericRikka]

Thanks for testing and for the quick feedback, @arturpzol — and thanks @barrkel for the encouragement. I’ve uploaded the current WIP patch as-is and I’m logging off for the night. Quick status:

• What it adds: an opt-in path to allow full RAW receives to cross the encryption boundary when replacing an existing dataset with -F: zfs send -w … | zfs receive -F -u --allow-encryption-change …
• How it’s plumbed: userspace long option --allow-encryption-change sets a user:allow_encryption_change=on receive prop which is consumed and stripped in the ioctl; kernel receives a boolean cookie and relaxes the boundary check only for full RAW + -F + opt-in. Incremental cross-boundary remains disallowed.
• Local verification:
  • Created encrypted src/fs, made @A, did zfs send -w src/fs@A | zfs receive -F -u --allow-encryption-change dst/fs.
  • Result: dst/fs is encrypted (encryption=aes-256-gcm, encryptionroot=dst/fs), feature@encryption=active.
  • Snapshot GUIDs match between src/fs@A and dst/fs@A.
  • After zfs load-key dst/fs + mount, file contents and hashes match.

0001-Applied-patch-and-corrected-an-error.patch

I have not cleaned up the patch or added zfstests yet — I just confirmed with the manual steps above. Tomorrow I’ll:

• split/clean the commits,
• add zfstests (success on full RAW -F + opt-in; fail without opt-in; fail on incremental cross-boundary; fail on pool without feature@encryption; replication with children), and
• open a PR with docs/manpage notes.

Feedback on the surface area (long option vs property name) or guard policy is very welcome. Thanks again!

[arturpzol]

@GenericRikka

Thank you for the new patch with the fixes.

I'm a bit confused - could you please confirm whether the following test scenario is expected to work?

truncate -s 200M /tmp/src.img
truncate -s 200M /tmp/dst.img
zpool create -f src /tmp/src.img
zpool create -f dst /tmp/dst.img
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase src/fs
echo ok > /src/fs/file
zfs snap src/fs@A

zfs create dst/fs
zfs send -w src/fs@A | zfs receive -uF --allow-encryption-change dst/fs

At the moment, this command fails with:

cannot receive new filesystem stream: invalid backup stream

Should this scenario already work with your patch - meaning that --allow-encryption-change should allow replacing an existing unencrypted dataset (dst/fs) with an encrypted RAW stream?

And if that’s the intended behavior, does the kernel side correctly recognize this new flag?

[GenericRikka]

@arturpzol I am very sorry for the confusion. Yes, the described scenario should work with this patch. At least it worked for me on my local git branch. But if you can not reproduce it, then either I did not fully fix this (only on my setup, but not on every setup), or the patch did not pick up all my changes (which I hope is the case). I will check and fix the patch as soon as I return home! Thanks again for the testing!

[GenericRikka]

Found the culprit. The difference between your testing protocol and my testing protocol was, that i did not create dst/fs manually, i just created dst, while you created dst/fs manually (with this i was able to reproduce the error you described). However, this still should not be a reason to fail and it most likely fails due to leftover option implementations i discarded, but did not fully clean up. Am working on it and will upload the patch as soon as I am done.

[almereyda]

This could also help with situations as in #12000 or #12649 and might benefit from #6824 or #15952.

#15687 also follows similar paths.

[GenericRikka]

Thanks for the cross-references, and sorry for the slow reply – I only just had time to look through these in detail.

My understanding of the related issues is roughly:

• #12000 shows what happens when the encryption hierarchy is manipulated via zfs send -Rw | zfs recv -d and something goes wrong: you end up with datasets that look fine on paper (encryption, encryptionroot, keystatus), but are no longer mountable. That’s a good example of how fragile things get once the encryptionroot metadata isn’t handled carefully.

• #12649 is essentially asking for the missing primitives: tooling to backup/restore keys and to manage encryption roots explicitly. That seems like the natural foundation for making operations such as “replace this existing dataset with a raw receive of an encrypted root” safe and supported instead of ad-hoc.

• #6824 and #15952 are more about key lifecycle and safety: multiple key slots / methods for unlocking a dataset, and a way to back up the master key to recover from a bad change-key or a malicious key change. Given that raw send/recv will happily propagate encryption metadata to backup targets, those concerns are very relevant to the kind of migration / re-homing workflow this issue is about.

• #15687 looks to be in the same “native encryption UX and metadata management” bucket as the above, even if the individual details differ.

From the user’s point of view, this issue is trying to capture one specific workflow:

“I have a pool with an encrypted root dataset, and I want to move that root (plus its inheriting children and snapshots) to a different vdev/pool layout using zfs send -wR | zfs receive -F, without re-encrypting and without rebuilding the hierarchy by hand.”

I agree that this overlaps heavily with the topics discussed in #12000 / #12649 / #6824 / #15952: a robust implementation here probably needs the same underlying capabilities to safely adjust encryption roots and handle key material, and it also needs to avoid creating the kind of broken hierarchies reported in #12000.

Even if the actual implementation work ends up happening under a more general “encryption roots & key management tooling” umbrella like #12649, I’d still be very happy to keep this concrete use-case tracked here, since “move an encrypted root dataset (and all its descendants) to new hardware or a resized pool” is a pretty common real-world operation.