Filesystem can not be mounted: Input/output error

[Fmstrat]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	20.04
Kernel Version	5.4.0-113-generic
Architecture	x86_64
OpenZFS Version	0.8.3-1ubuntu12.13

Describe the problem you're observing

After a reboot, ZFS volumes aren't mounting, some do, some don't.

# zpool status s -v
  pool: s
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 1 days 04:22:08 with 0 errors on Mon May  9 04:46:19 2022
config:

        NAME                                   STATE     READ WRITE CKSUM
        s                                      ONLINE       0     0     0
          raidz2-0                             ONLINE       0     0     0
            ata-ST16000NM001G-<redacted>  ONLINE       0     0     0
            ata-ST16000NM001G-<redacted>  ONLINE       0     0     0
            ata-ST16000NM001G-<redacted>  ONLINE       0     0     0
            ata-ST16000NM001G-<redacted>  ONLINE       0     0     0
            ata-ST16000NM001G-<redacted>  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        s/storage/n/random:<0x0>
        s/storage/n/backup:<0x0>
        s/storage/n/plex:<0x0>
        s/storage/n/[email protected]:<0x0>
        s/storage/n/security:<0x0>
        s/storage/n/receipts:<0x0>
        s/storage/n/creative:<0x0>
        s/storage/n/music:<0x0>
        s/storage/n/ebooks:<0x0>
        s/storage/n/pictures:<0x0>
        s/storage/n/software:<0x0>

# zfs mount s/storage/n/random
filesystem 's/storage/n/random' can not be mounted: Input/output error
cannot mount 's/storage/n/random': Invalid argument

The issue with the snapshot above was not visible at boot. It only came up after I tried to mount it. I was able to mount the original snapshot (non-incremental) with mount -t zfs s/storage/n/[email protected] /mnt/test but I don't have original snapshots for every volume.

Have I lost this data or is this recoverable? What works vs what doesn't seems eerily similar to https://github.com/openzfs/zfs/issues/8103

Describe how to reproduce the problem

This happened after the filesystem locked up the server. Commands accessing the zpool stopped responding, and required a forced reboot (kill -9 didn't work).

[Fmstrat]

Some notes that may help:

• These volumes use native encryption
• They were sent raw from another server in another state
• Incremental snaps have been sent just fine up until now

[Fmstrat]

After attempting zfs mount s/storage/n/random

zfs events -v

May 29 2022 10:22:58.641738494 ereport.fs.zfs.authentication
        class = "ereport.fs.zfs.authentication"
        ena = 0x222d49cc46000001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x6c9a0324e11e4bcb
        (end detector)
        pool = "s"
        pool_guid = 0x6c9a0324e11e4bcb
        pool_state = 0x0
        pool_context = 0x0
        pool_failmode = "wait"
        zio_objset = 0x203
        zio_object = 0x0
        zio_level = 0xffffffffffffffff
        zio_blkid = 0x0
        time = 0x629381c2 0x264026fe 
        eid = 0x5e

zdb -ddddddddddd s/storage/n/random 0

Dataset s/storage/n/random [ZPL], ID 515, cr_txg 55, 10.2G, 1341 objects, rootbp DVA[0]=<0:4561eb61c000:3000> DVA[1]=<0:3dac6cd0000:3000> [L0 DMU objset] fletcher4 uncompressed authenticated LE contiguous unique double size=1000L/1000P birth=6786579L/6786579P fill=1341 cksum=1a382de2b3:472f3f2d19c6:678aeb2dae11b8:6ab5b1b595ebfe13

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         0    6   128K    16K  1.94M     512   784K   85.52  DMU dnode (K=inherit) (Z=inherit)
        dnode flags: USED_BYTES 
        dnode maxblkid: 48
                (object encrypted)
Indirect blocks:
               0 L5      0:453ad05fa000:3000 0:417465d5000:3000 20000L/1000P F=1341 B=6786579/6786579 cksum=38f75516ecac3b9:6245ebc924f526fe:bc21378ec23acb05:cffb65f15cd1ae3b
               0  L4     0:4561eb619000:3000 0:3dac6cb2000:3000 20000L/1000P F=1341 B=6786579/6786579 cksum=39708c0200a872c:6c315aed5d9f5b31:d7a33c44e21b4277:2d4c602a8ab3294d
               0   L3    0:4561eb616000:3000 0:3dac6caf000:3000 20000L/1000P F=1341 B=6786579/6786579 cksum=391d61952c436bc:655f4d4c3c1837a3:f3afa6884b69f1fb:15c210dbae408617
               0    L2   0:4561eb613000:3000 0:3dac6cac000:3000 20000L/1000P F=1341 B=6786579/6786579 cksum=39904c9b205bd0c:6ec8be242f9b78a1:7a2ade7a821ca3a6:7cc890856018194f
               0     L1  0:44d02ce56000:3000 0:200b15ef1000:3000 20000L/1000P F=1341 B=6786579/6786579 cksum=46dc39d5f67fce1:4f49187ee197b77d:c888e98fdcfb7005:bc148888941aacfd
               0      L0 0:453ad0465000:9000 0:417466d7000:9000 4000L/4000P F=31 B=6786579/6786579 cksum=c5205989ca625345:2b8f66c7c9a31982:94f9158f6ca88b4b:835486f8dd072e63
            4000      L0 0:453ad046e000:9000 0:41746629000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=b25bbc73112ad62f:3f1f95d6d13044ce:4664d20c782dcd6c:c7ea403149cae970
            8000      L0 0:453ad0480000:9000 0:417465b1000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=eb6d2fea3a9c3414:8b2d5f00b062aeb9:ee1f2cbbf5f631e5:64b1915585145a2d
            c000      L0 0:453ad0477000:9000 0:41746632000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=ed4d5987fbdeae1a:ef98a8b9e3c58787:5eb5e893035171e2:5b774baf955d95d4
           10000      L0 0:453ad0489000:9000 0:417465ba000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=ef13b18e094645cf:1a9bc43d6b4c07f4:bf115fcc52c29114:40a9a95d6422fe42
           14000      L0 0:453ad0492000:9000 0:417465c3000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f11581dff59e07cf:4cf041c94dcc88b2:54fcfe0aed4463be:672cc4ff6a502d55
           18000      L0 0:453ad049b000:9000 0:417465cc000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=ecc5e844bd6cf15a:f4745c049f735e8d:78634f80d8413d9f:28452c8294ea8cc0
           1c000      L0 0:453ad04a4000:9000 0:417466f2000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=eee0547168cee2b9:22a9e61add50c27d:bcf58f46f78bae:1b5a9f1568b98458
           20000      L0 0:453ad04ad000:9000 0:417466fb000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f3d6be807d990923:fa4994034f061ea8:1c5d743a12e3c631:ffbc922d2ab22754
           24000      L0 0:453ad04bf000:9000 0:4174670d000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f0eeb9613cee306f:90b9e46869f29412:f39e54fc29632dbe:7dee39e5702cc490
           28000      L0 0:453ad04b6000:9000 0:41746704000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=eda333d1bef45d59:911a3ac84202526d:62c26be8b8b40d96:f72b540c45323a43
           2c000      L0 0:453ad04c8000:9000 0:41746716000:9000 4000L/4000P F=22 B=6786579/6786579 cksum=e61651c0694c120d:f1557b6c20529d5f:e32d71b7441ecc1b:6a0e375968f44971
           30000      L0 0:453ad04d1000:9000 0:4174671f000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f035415cca952edf:2a39dde8746300ea:d0eb0e09fe71f90a:1f72161aac43dc0
           34000      L0 0:453ad04da000:9000 0:41746728000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=edcae67009cb85fc:575b64474351f3ae:fb5ca0bf63141c05:244530c5f03871a6
           38000      L0 0:453ad04e3000:9000 0:41746731000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=ef9eaacba96a0bb5:579be615bc0b6e90:167a1fa64e18ff9e:9010b8eadc46609d
           3c000      L0 0:453ad04ec000:9000 0:4174673a000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f0d8a76000e230e8:ffae8220859038ec:85be61e693fffac9:82d44f63449196eb
           40000      L0 0:453ad04f5000:9000 0:41746743000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f2b2bf0c688a432c:513bbddb3b0695c0:3c1f0b7fafa41713:888cc7ad2c2aeea3
           44000      L0 0:453ad04fe000:9000 0:4174674c000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f2b853f5f94bb729:87987b23a74f8580:da69a9db1ab8edd9:4478ffec3415cfa2
           48000      L0 0:453ad0507000:9000 0:41746755000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f1907c966c4429ce:fa3f0443e7030107:76964df9917526a4:6031289b12008b92
           4c000      L0 0:453ad0510000:9000 0:4174675e000:9000 4000L/4000P F=21 B=6786579/6786579 cksum=e30644864c02c163:19e3c9489e035358:1abf19336bb9ea95:dd1ff01480397f63
           50000      L0 0:453ad0519000:9000 0:41746767000:9000 4000L/4000P F=30 B=6786579/6786579 cksum=d37d02d7b1f2ccde:6b807a1437247393:e5fc1b2af8da8b46:f81edecf6b1115f1
           54000      L0 0:453ad0522000:9000 0:41746770000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=e660d0767699bdae:5317b242cdfe322c:2f145a6ad16fea0c:7bd9f82207a5fff
           58000      L0 0:453ad052b000:9000 0:41746779000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=efc135451c38c820:71803c7e6cb853e4:d2e4c6ee6f1d079a:360a50bfb9005048
           5c000      L0 0:453ad0534000:9000 0:41746782000:9000 4000L/4000P F=19 B=6786579/6786579 cksum=dcb1a3d2dca91d87:142a71787aee81c6:e345a9f1162e45b0:70107d7b79908b75
           60000      L0 0:453ad053d000:9000 0:4174678b000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=e6906c08d7771693:9ca5ff7bd3b875ce:fd98191441dfba4e:13e638a0bbab2925
           64000      L0 0:453ad0546000:9000 0:41746794000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=eb6075148c780709:ca7a3481f1f9f9e4:96b959b77e34ce43:eb1598544a48e1ca
           68000      L0 0:453ad054f000:9000 0:4174679d000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=eb3f99c88120f4c9:a006fcb233182494:d4d7a6fedbcf6617:d1cc0accde59987d
           6c000      L0 0:453ad0558000:9000 0:417467a6000:9000 4000L/4000P F=25 B=6786579/6786579 cksum=f3a2c933ba40bc57:fe1b77584d3d9835:9aa6bb045ca952a8:cce271ecf9a14853
           70000      L0 0:453ad056a000:9000 0:417467b8000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=ec7cfc09883224b8:d6461dd5904e300:9f30f1fa192025ce:8075bc9aa019a795
           74000      L0 0:453ad0561000:9000 0:417467af000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f4984833e9d88527:178babc01de4a95b:194cc723c2eb0634:35a531012d19bc69
           78000      L0 0:453ad0573000:9000 0:417467c1000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f141f55cfb8ae45f:ecf65326ebc1d533:1b0236101a26b9f5:337399292eb01a58
           7c000      L0 0:453ad057c000:9000 0:417467ca000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f13a7d15ecebaf88:ce3ec01d947ed450:bbcefece9edd970c:a05818074eaa429b
           80000      L0 0:453ad0585000:9000 0:417467d3000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=e5aae880e3f48927:8ff07f3beeb1ebeb:4f0ab661443e6bc2:60f52b9d2d1c125
           84000      L0 0:453ad058e000:9000 0:417467dc000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=e50710b8dba20a5c:9a440c179e79fa86:5a63b8ebd4abef7e:7408543f35cdf887
           88000      L0 0:453ad0597000:9000 0:417467e5000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=eefa59d52d0e0c93:f246ecd9d4e3a298:2789b3705fa22ac:4ec86aecfb4d3d0d
           8c000      L0 0:453ad05a0000:9000 0:417467ee000:9000 4000L/4000P F=22 B=6786579/6786579 cksum=f016d48464251c54:e6a8554fa5e64aea:53baf589b0fc8f34:9240eb11793fe3ef
           90000      L0 0:453ad05a9000:9000 0:417467f7000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=ebf247601ea5ebb4:16879dc10488a3cb:5759cec35445993:bd00c95c8f092fc8
           94000      L0 0:453ad05b2000:9000 0:41746800000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f3121d3447b0d127:5e82a1b074fdfaff:4fe8e74d2962d44b:11ccc4eef8f107ef
           98000      L0 0:453ad05bb000:9000 0:41746809000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=f2941b2f3ddc96d2:85570474f8165e98:55da246b7d675be2:ce8e5b912a31dd38
           9c000      L0 0:453ad05c4000:9000 0:41746812000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=efa361125f1f4b4f:31e8a799edc7c3db:938bb506c5e98a61:8e89811377f96ba1
           a0000      L0 0:453ad05cd000:9000 0:4174681b000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=ec3e8bf9d264b28e:49da394a56b8e225:dff696afd478acd6:8de24efb77fd1924
           a4000      L0 0:453ad05d6000:9000 0:41746824000:9000 4000L/4000P F=22 B=6786579/6786579 cksum=eb77349c8842a11a:e9fc8fc5474ed970:b1f6b0616d64ca3:238bd4e72479f460
           b0000      L0 0:453ad05df000:9000 0:4172508f000:9000 4000L/4000P F=32 B=6786579/6786579 cksum=edf2a642120f7bf3:7c0857947cb2c9a1:987d6fb173bf05bd:9687dac1706c45bb
           b4000      L0 0:453ad05e8000:9000 0:41725098000:9000 4000L/4000P F=24 B=6786579/6786579 cksum=ee653458c731afb4:5575bade2241e733:3e4b8fd32c099fac:82d39605cc32cc10
           c0000      L0 0:453ad05f1000:9000 0:417250a1000:9000 4000L/4000P F=5 B=6786579/6786579 cksum=616ed2da495882a3:9956d736f67d25f6:6f3d8d0e8931bc67:c22c5091d5f4e145

                segment [0000000000000200, 000000000002ec00) size  186K
                segment [0000000000030000, 000000000004ea00) size  122K
                segment [0000000000050000, 0000000000050e00) size 3.50K
                segment [0000000000051200, 000000000005e600) size   53K
                segment [0000000000060000, 000000000006f200) size 60.5K
                segment [0000000000070000, 000000000008ec00) size  123K
                segment [0000000000090000, 00000000000a6c00) size   91K
                segment [00000000000b0000, 00000000000b7000) size   28K
                segment [00000000000c0000, 00000000000c0a00) size 2.50K

[Fmstrat]

@tcaputi hope you don't mind the tag, apologies if you do. I just know you've been through this one before.

[rincebrain]

Well, you're running a 2+ year old version with known bugs (not that the latest version of native encryption doesn't also have a host of known bugs nobody's fixed), so perhaps you should consider not doing that.

That said, my wild guess would be that it's the unfixed bug where incremental receive will sometimes happily do zfs change-key incorrectly and update the metadata to think the wrong wrapping key is the one in use, which you only notice on reboot...when it tries to unlock the dataset, and because it has the incorrect idea of which key to use, it obviously does not succeed.

e: #12000, #12614

[Fmstrat]

@rincebrain Thank you! So it looks like in Ubuntu 20.04 I can update from 0.8.3-1ubuntu12.13 to 0.8.3-1ubuntu12.14. If this is indeed it, then I think this may have occurred due to some initial issues during transfer. Here's what happened:

• I made s/storage on the backup server as the encryption root. I then added a bunch of other pools of s/storage/* underneath it
• I used send to put s/storage and all sub-pools onto external drives
• I drove those drives to the main server
• A couple of the sub-pools had failed files, so all of s/storage wouldn't send from the external to the main server. I had to send each sub-pool independently, likely assigning the encryption key to each vs inheriting from the root
• Incremental snaps were then sent from the main server to the backup server
• This is likely the first time I've rebooted the backup server

A few questions:

• Is this update sufficient if I do it on both the main server and the backup server?
• Would upgrading allow me to mount, or are there some steps I would need to take first?
• I.E. could I assign an encryption key to each of the sub-pools manually, then mount (these are mounted read-only always on the backup server)?

Just trying to figure out the best way to get up and running again under Ubuntu 20.04. Thanks!

[rincebrain]

The current series of releases is 2.1.x, you are on 0.8.x. Upgrading from 0.8.3-mumble13 to 0.8.3-mumble14 was not my suggestion, though you could always try reporting your issue to Ubuntu's bugtracker in that case, as I would be pretty astonished if 0.8.x saw any fixes or releases ever again here.

Upgrading things won't fix this problem at this point. If I'm right about what happened, it wouldn't have even prevented them before, since that bug is unfixed.

One could write code to rekey the individual datasets forcibly ignoring that you don't have the old key loaded, there's even the framework for it to exist already, it's just not written anywhere (that I'm aware of); I have an incomplete version that doesn't work on every key type at the moment, but haven't gone back to figure out what's missing in a bit.

[Fmstrat]

@rincebrain Well that doesn't sound promising then. What's your recommended upgrade methodology to 2.1.x for Ubuntu 20.04 that wouldn't involve me losing access to data on the main server? They're supposed to still be supporting 20.04, but it sounds like zfs is woefully out of date there.

I assume by "write code" you don't mean use the zfs command to assign a new key? I'm not sure I'd have the expertise to pull that off without a guide or walk-through of some kind.

Is there any way I can get this to work without starting from scratch? Such as leveraging the first snap that is mountable in combination with an upgrade (from.. somewhere)? This is many TB of data separated by many states, so I can't easily do full sends across the internet.

[Fmstrat]

From this reddit thread it looks like I could upgrade without issue, and from this one there appears to be a maintained PPA.

So if I went that route, upgraded on both systems, would it be possible to roll back to a working snapshot (if I have one) on the backup server, then send from the main server to get things in sync? Or do I need to send from scratch all over again? And since that's Reddit, and you're you, does the above upgrade path even make sense to a ZFS dev? Thanks again!

[rincebrain]

I believe there's a popular PPA; you could also build packages from source; that's just general advice though, if I'm correct about the problem, it's still unfixed now too, so it wouldn't have helped.

I don't, no. If you can mount some old snapshots, though, but not newer ones, on the same dataset, I don't think it can be the bug I'm describing, because I don't think there's any sort of old key history kept, so they should all be equally unusable.

One thing you could try, potentially, assuming it's one of the encryption bugs that has been fixed instead, would be to upgrade to 2.1.4 using the PPA or building your own packages or w/e, import the pool, see if you can mount things, and if not, you could try doing a raw send of the encrypted dataset and receiving it anew on the same pool somewhere else, e.g. zfs send -w z/storage@foo | zfs recv z/storage_new, and seeing if you can unlock and mount it afterward,, since there are one or two bugs that would work for.

[Fmstrat]

Ahh yes. I can mount the original snapshots, just like in the first thread I linked. Every incremental snapshot that was sent after won't mount, but the first will. And any pool that only has that first snapshot also mounts as expected. A big concern I have right now is if the main server reboots and I lose access to my data altogether.

[Fmstrat]

Btw, this is what I mean by they are different from a key perspective:

Main server:

# zfs get keylocation s/storage/n/random
NAME                PROPERTY     VALUE               SOURCE
s/storage/n/random  keylocation  file:///etc/zfskey  local

Backup server:

# zfs get encryption s/storage/n/random
NAME                PROPERTY    VALUE        SOURCE
s/storage/n/random  keylocation  none         default

The backup server was inheriting from s/storage originally. But maybe it's not anymore due to the odd sends from the other machine?

I.E.:

• Backup server has s/storage/n/random original snap with inherited encryption key from s/storage
• That snap is sent to the main server, but a key is assigned directly vs inherited
• Main server makes changes to the filesystem
• Main server sends an incremental s/storage/n/random to the backup server
• At this point, the original snap can mount, the new one can't

Would the backup server get confused without keylocation set at this point?

[rincebrain]

Yes, that is certainly quite confused, but I think I'm surprised that it can still mount the old one, as like I said, changing the wrapping key or no, the actual key that encrypts the data has remained the same, so you should still need the same unlock for both cases, AFAIK.

[Fmstrat]

Ok, thanks. If you were in my boat, what steps would you take?

[rincebrain]

I'd probably try the send/recv thing I mentioned above, and if that didn't work, go hackily patch the code to override that property value in that specific case to be what you know to be correct and/or finish the WIP code for forcibly resetting the property.

I'm taking a break for a bit after I spent a bunch of time today hacking on something complicated and then someone pointed out I was being foolish and there was a much simpler solution, but if you remind me later I'll see if I can't make that branch that's incomplete for fixing this complete and then you can try it.

[AttilaFueloep]

If you can mount some old snapshots, though, but not newer ones, on the same dataset, I don't think it can be the bug I'm describing, because I don't think there's any sort of old key history kept, so they should all be equally unusable.

Yes, I agree. This looks more like the user accounting MAC issue which was fixed in #12981 a couple of months ago. So I'd say a recent ZFS version (2.1.4 or git@master) should be able to mount the datasets, at least it's worth a try.

OTOH if it's really an issue with overwriting the wrapping key, it would be good to see the exact sequence of commands which got you into this situation. Unfortunately, from the description you gave I can't deduce the exact commands you issued. I've a couple of reproducers for key issues which I can compare your commands against.

With "mount the snapshot" you mean mount -t zfs pool/dataset@snap /mnt, right?

[rincebrain]

I think #12981 only means you could send/recv and then unlock the resulting recv, not mount the existing dataset, though I think writing a zhack command to trigger the same behavior without a recv might be reasonably doable...

[Fmstrat]

I will give some of this a shot, and also try to recreate the issue before (and after) I update.

If I need to fully re-send, should I be using raw, or re-encrypting at destination?

[rincebrain]

Re-encrypting at destination would avoid problems with change-key propagating surprisingly, but there are other bugs which only require receiving raw no matter what (hi #11679), so pick your poison, IMO.

[AttilaFueloep]

@rincebrain Honestly I can't tell, didn't look very thoroughly, sorry. This issue just reminded me of an issue a had some time ago where I couldn't mount a raw send/recvd encrypted dataset (without key manipulation) while I could mount an older snapshot. It was due to the MAC of the accounting object failing and it did show the Permanent errors dataset: <0x0> symptom.

@Fmstrat Depends on your use case. If you want to do raw incremental sends, re-encrypting isn't an option. If you do raw sends, I'd strongly encourage you to not manipulate the keys on the receiving side since there are some edge cases where you could end up with inaccessible received datasets. If the send communication channel is trusted OTOH, you can re-encrypt plain incremental sends.

[AttilaFueloep]

And of course what @rincebrain just wrote.

[rincebrain]

@AttilaFueloep i wonder if the root cause of that is actually related to the #11679 panic, which it turns out is caused by the accounting code's dirtying every object and syncing out trick causing inconsistent state in memory and a NULL deref if someone tries to read the dnode at the wrong time

[Fmstrat]

@AttilaFueloep what exactly do you mean by "manipulate the keys"? I'm trying to understand if this would help my current situation and if so, how.

[Fmstrat]

@AttilaFueloep @rincebrain

Ok, before I try anything else, I have this fully reproducible in 0.8.3-1ubuntu12.13:

# Make server A, create snapshot 1, then make that pool read-only
truncate -s 1G /tmp/tmpa.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmpa /tmp/tmpa.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmpa/storage tmpa/storage
zfs create -o mountpoint=/mnt/tmpa/storage/test tmpa/storage/test
echo 1 > /mnt/tmpa/storage/test/testfile.txt
zfs snap -r tmpa/storage@1
zfs set readonly=on tmpa/storage/test

# Make server B, send from A to B
truncate -s 1G /tmp/tmpb.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmpb /tmp/tmpb.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmpb/storage tmpb/storage
zfs send -w tmpa/storage/test@1 |pv -Wbraft |zfs recv tmpb/storage/test

# Set server B writable, change file, send to A
zfs set readonly=off tmpb/storage/test
zfs set mountpoint=/mnt/tmpb/storage/test tmpb/storage/test
zfs set keylocation=file:///etc/zfskey tmpb/storage/test
zfs load-key tmpb/storage/test
zfs mount tmpb/storage/test
echo 2 > /mnt/tmpb/storage/test/testfile.txt
zfs snap tmpb/storage/test@2
zfs send -w -I tmpb/storage/test@1 tmpb/storage/test@2 |pv -Wbraft |zfs recv tmpa/storage/test

# Unmount and try to remount A. Failure
zfs unmount tmpa/storage/test
zfs mount tmpa/storage/test

# Cleanup
zpool destroy tmpa
zpool destroy tmpb
rm /tmp/tmpa.img
rm /tmp/tmpb.img

Given this, would you recommend me try anything else before I upgrade?

[Fmstrat]

I did the above on a spare laptop with Ubuntu 20.04 on it, then upgraded to this version from the PPA:

zfs-2.1.4-0york0~20.04
zfs-kmod-2.0.2-1ubuntu5.4

However I'm still unable to mount, so no luck there. I then tried re-sending the incremental in hopes I could salvage:

# zfs send -w -I tmpb/storage/test@1 tmpb/storage/test@2 |pv -Wbraft |zfs recv tmpa/storage/test
11.0KiB 0:00:00 [ 380KiB/s] [ 380KiB/s]

But still no luck on mounting afterwards. For the heck of it, I reran the above test from scratch in 2.1.4 and lo and behold, the issue exists there, too.

[Fmstrat]

So far the only way I can get this to work is by re-sending the entire pools from scratch and setting them up with their own encryption roots:

# Make server A, create snapshot 1, then make that pool read-only
truncate -s 1G /tmp/tmp2a.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2a /tmp/tmp2a.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2a/storage tmp2a/storage
zfs create -o mountpoint=/mnt/tmp2a/storage/test tmp2a/storage/test
echo 1 > /mnt/tmp2a/storage/test/testfile.txt
zfs snap -r tmp2a/storage@1
zfs set readonly=on tmp2a/storage/test

# Make server B, send from A to B
truncate -s 1G /tmp/tmp2b.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2b /tmp/tmp2b.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2b/storage tmp2b/storage
zfs send -w tmp2a/storage/test@1 |pv -Wbraft |zfs recv tmp2b/storage/test

# Set server B writable, change file, send to A
zfs set readonly=off tmp2b/storage/test
zfs set mountpoint=/mnt/tmp2b/storage/test tmp2b/storage/test
zfs set keylocation=file:///etc/zfskey tmp2b/storage/test
zfs load-key tmp2b/storage/test
zfs mount tmp2b/storage/test
echo 2 > /mnt/tmp2b/storage/test/testfile.txt
zfs snap tmp2b/storage/test@2
zfs send -w -I tmp2b/storage/test@1 tmp2b/storage/test@2 |pv -Wbraft |zfs recv tmp2a/storage/test

# Unmount and try to remount A. Failure
zfs unmount tmp2a/storage/test
zfs mount tmp2a/storage/test

# Destroy server A test pool and resend from scratch
zfs destroy -r tmp2a/storage/test
zfs send -w tmp2b/storage/test@2 |pv -Wbraft |zfs recv tmp2a/storage/test
zfs set readonly=on tmp2a/storage/test
zfs set mountpoint=/mnt/tmp2a/storage/test tmp2a/storage/test
zfs set keylocation=file:///etc/zfskey tmp2a/storage/test
zfs load-key tmp2a/storage/test
zfs mount tmp2a/storage/test

# Make a change on server B and send
echo 3 > /mnt/tmp2b/storage/test/testfile.txt
zfs snap tmp2b/storage/test@3
zfs send -w -I tmp2b/storage/test@2 tmp2b/storage/test@3 |pv -Wbraft |zfs recv tmp2a/storage/test

# Unmount and try to remount A. Success
zfs unmount tmp2a/storage/test
zfs mount tmp2a/storage/test

# Cleanup
zpool destroy tmp2a
zpool destroy tmp2b
rm /tmp/tmp2a.img
rm /tmp/tmp2b.img

[Fmstrat]

One more test. The error also occurs if I send the entire pool, instead of just "test":

# Make server A, create snapshot 1, then make that pool read-only
truncate -s 1G /tmp/tmp2a.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2a /tmp/tmp2a.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2a/storage tmp2a/storage
zfs create -o mountpoint=/mnt/tmp2a/storage/test tmp2a/storage/test
echo 1 > /mnt/tmp2a/storage/test/testfile.txt
zfs snap -r tmp2a/storage@1
zfs set readonly=on tmp2a/storage/test

# Make server B, send from A to B
truncate -s 1G /tmp/tmp2b.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2b /tmp/tmp2b.img
# -=> First difference, send the entire pool recursively
#zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2b/storage tmp2b/storage
#zfs send -w tmp2a/storage/test@1 |pv -Wbraft |zfs recv tmp2b/storage/test
zfs send -wR tmp2a/storage@1 |pv -Wbraft |zfs recv tmp2b/storage

# Set server B writable, change file, send to A
zfs set readonly=off tmp2b/storage/test
zfs set mountpoint=/mnt/tmp2b/storage/test tmp2b/storage/test
# -=> Next difference, key location and load keys for storage
#zfs set keylocation=file:///etc/zfskey tmp2b/storage/test
#zfs load-key tmp2b/storage/test
zfs set keylocation=file:///etc/zfskey tmp2b/storage
zfs load-key tmp2b/storage
zfs mount tmp2b/storage/test
echo 2 > /mnt/tmp2b/storage/test/testfile.txt
zfs snap tmp2b/storage/test@2
zfs send -w -I tmp2b/storage/test@1 tmp2b/storage/test@2 |pv -Wbraft |zfs recv tmp2a/storage/test

# Unmount and try to remount A. Failure
zfs unmount tmp2a/storage/test
zfs mount tmp2a/storage/test

# Cleanup
zpool destroy tmp2a
zpool destroy tmp2b
rm /tmp/tmp2a.img
rm /tmp/tmp2b.img

[Fmstrat]

Sorry for the flood, just trying to eliminate dead-ends. Using recursive snap doesn't seem to be the issue. It seems to be related to when the encryption root exists in tmpa/storage and is inherited in tmpa/storage/test but tmpb/storage/test is an encryption root.

# Make server A, create snapshot 1, then make that pool read-only
truncate -s 1G /tmp/tmp2a.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2a /tmp/tmp2a.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2a/storage tmp2a/storage
zfs create -o mountpoint=/mnt/tmp2a/storage/test tmp2a/storage/test
echo 1 > /mnt/tmp2a/storage/test/testfile.txt
# -=> Just snap test
#zfs snap -r tmp2a/storage@1
zfs snap tmp2a/storage/test@1
zfs set readonly=on tmp2a/storage/test

# Make server B, send from A to B
truncate -s 1G /tmp/tmp2b.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2b /tmp/tmp2b.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2b/storage tmp2b/storage
zfs send -w tmp2a/storage/test@1 |pv -Wbraft |zfs recv tmp2b/storage/test

# Set server B writable, change file, send to A
zfs set readonly=off tmp2b/storage/test
zfs set mountpoint=/mnt/tmp2b/storage/test tmp2b/storage/test
zfs set keylocation=file:///etc/zfskey tmp2b/storage/test
zfs load-key tmp2b/storage/test
zfs mount tmp2b/storage/test
echo 2 > /mnt/tmp2b/storage/test/testfile.txt
zfs snap tmp2b/storage/test@2
zfs send -w -I tmp2b/storage/test@1 tmp2b/storage/test@2 |pv -Wbraft |zfs recv tmp2a/storage/test

# Unmount and try to remount A. Failure
zfs unmount tmp2a/storage/test
zfs mount tmp2a/storage/test

# Cleanup
zpool destroy tmp2a
zpool destroy tmp2b
rm /tmp/tmp2a.img
rm /tmp/tmp2b.img

[AttilaFueloep]

@rincebrain Your last comment there was an interesting read. Look like the integration of encryption with the quota accounting still has some rough edges. Maybe I can say more after replaying the given reproducers.

[AttilaFueloep]

@Fmstrat

what exactly do you mean by "manipulate the keys"

Well, running zfs change-key [-i] on the received datasets or changing encryption roots. In short anything which will change the wrapping key on the destination. Sorry, can't give much detail right now, it's been a while I analyzed this and my memories are faint. Will have to lookup my notes on this to say more.

[AttilaFueloep]

Nice to see some reproducers. It's quite late here already, I'll try to digest your input and reproduce it on a somewhat current master tomorrow.