Cryptsetup toolstack version bump + reencryption cleanup (LUKSv2+Luksv1 proper support + reencryption on Q4.2 + BTRFS dual LUKS containers install)

[tlaurion]

trafficstars

• Bump kernel from 5.10.5 -> 5.10.214

  • Cloudfare patches to speed up LUKS encryption were upstreamed into linux kernel and backported to 5.10.9+.
    • See https://github.com/cloudflare/linux/issues/1#issuecomment-763085915
  • cryptsetup2 toolstack version bump and script fixes to support multi-LUKS containers (BTRFS QubesOS 4.2)
    • Argon2 used externally and internally: requires a lot of RAM and CPU to derivate passphrase to key validated in key slots.
      • This is used to rate limit efficiently bruteforcing of LUKS key slots, requiring each offline brute force attempt to consume ~15-30 seconds per attempt
      • Of course, strong passphrases are still recommended, but bruteforcing LUKSv2 containers with Argon2 would require immense time, ram and CPU even to bruteforce low entropy passphrase/PINs.
    • passphrase change doesn't reuse LUKS key slot anymore: cryptsetup passhprase change enforces key slot rotattion (new one consusumed per hot operation: the old one wiped internally after change. EG: LUKS key slot 1 created, then 0 deleted)
    • reencryption doesn't permit old call arguments. No more direct-io; inadmissibly slow through AIO (async) calls, need workarounds for good enough perfs (arguments + newer kernel with cloudfare fixes in tree)
  • cryptsetup required whole toolstack version bump:
    • cryptsetup requires lvm2 2.03.23+
    • cryptsetup requires libaio, which is also included in this PR (could be hacked out but deep dependency at first sight: left in, deprecates all legacy boards)
      • which requires util-linux 2.39
    • patches for reproducible builds are included for above 4 packages.

• luks-functions was updated to support the new cryptsetup 2.6.1 version calls/changes as opposed to old 2.3 in master

  • reencryption happen in direct-io, offline mode and without locking, requiring linux 5.10.9+ to bypass linux queues
    • from tests, this is best for performance and reliability in single-user mode, as before, op cannot be interrupted.
  • LUKS container ops now validate Disk Recovery Key (DRK) passphrase and DRK key slot prior of going forward if needed, failing close early.
    • Heads don't expect DRK to be in static key slot anymore, and finds and keeps track of the used DRK key slot dynamically.
    • If reencrytipn/passphrase change: make sure all LUKS containers on same block device can be unlocked with same DRK
      • Reencryption: requires to know which key slot to reencrypt.
        • Find LUKS key slot that unlocks with DRK passphrase unlock prior of reencrypt call
      • Passphrase change: no slot can be passed, but key slot of DRK rotates.

• kexec-seal-key

  • TPM LUKS Disk Unlock Key key slots have changed to be set in max slots per LUKS version (LUKSv1:7 /LUKSv2: 31)
    • If key slot != default LUKS version's keyslot outside of found DRK key slot: prompt the user before wiping that key slot, otherwise wipe automatically
      • This takes for granted that the DRK key slot alone is needed on the system and Heads solely controls the LUKS key slots.
        • If user has something else going on, ie: Using USB Security dongle + TPM DUK, then the user will need to say no when wiping additional key slot not being DRK or LUKSv1:7/LUKSv2:31.
          • It was suggested to leave LUKS key slots outside of DRK alone, but then: what to do when all key slots would be used on next op?
            • Alternative implementation could be to only prompt users to wipe key slots other then DRK when key slots are all used (LUKSv1: 0-7, LUKSv2: 0-31)
              • But then cleanup would need to happen prior of operations (LUKS passphrase change, TPM DUK setup) and could be problematic, where OS might fail (QubesOS DRK util doesn't do test on this, not sure why any other user util would also check for that corner case. Was decided against.)
  • LUKS containers now checked to be same LUKS version prior of permitting to set TPM DUK and will refuse to go forward if different versions of LUKS found across LUKS containers specified to be unlocked with TPM DUK, even if across different block devices.

---

TODOs:

• async (AIO) calls are not used. direct-io is used instead. libaio could be hacked out
  • this could be subject to future work.

Notes:

• time to deprecated legacy boards the do not enough space for the new space requirements
  • x230-legacy, x230-legacy-flash, x230-hotp-legacy
    • t430-legacy, t430-legacy-flash, t430-hotp-legacy were already deprecated. No more legacy board support
  • None of the legacy boards are now built by CircleCI. Legacy boards are dead, long lived legacy boards.

Unrelated:

• typos fixes found along the way

---

OLD: I finally got a grip on where stems the problem discussed under https://github.com/linuxboot/heads/issues/1539

• ~cryptsesup requires async/sync ops from kernel (removed directio ops)~
  • cryptsetup uses direct-io when in offline mode without locking which is new calls changed in luks helper script
• libaio is new strong dependency of lvm2 (hacking needed on lvm2 side to remove dep)
• ~kernel AIO needed otherwise warning (lets see if that is verbose without being under debug mode later)~
• cryptsetup requires newer libdevmapper and dmsetup to deal also with that
• those are provided by newer lvm2 binaries, including blkid and dmsetup itself
  • which required newer util-linux version to provide such

Todo:

• [x] Make sure kernel crypto backend requirements are as small as needed
• [x] Review patches and clean them
• [x] Removeinitrd/test_reencrypt_ram.sh when ramfs raw disk reencryption meets normal speed
• [x] test on real hardware
• [x] legacy boards now deprecated. refactoring of /etc/ash_functions can now occur to depend only on bash
• [ ] documentation for deprecation of legacy board
• [x] open other issues, including newer distro impossibilities to having fs reencrypted if done through cryptsetup 2.6.1
• [x] cloudfare optimizations down from cryptsetup calls https://github.com/linuxboot/heads/issues/1539#issuecomment-1832412784 still needed?

---

• [ ] Disclose publicly needed firmware upgrade/kernel commit bump downstream (that having gone unnoticed confirms that noone reencrypted Q4.2/Q4.2.1 installation up to me discovering this. I can only repeat this, but OEM not pushing users to reencrypt their encrypted drives from OEM means that a malicious worker could backup LUKS header for all laptops where OSes are preinstalled, and sell that LUKS header backup at high prices for daily used laptops theft where OEM provisioned DRK passphrase can then be reused to access FDE content supposedly protected by encryption. The only way to completely transfer OEM ownership of a laptop to the end user is to rerun the Re-Ownership wizard and accept reencrypting the disk. DRK != DRK passphrase. Changing the DRK passphrase won't permit a LUKS header backup/restore from permitting old DRK passphrase from decrypting the FDE content.

---

Old branch prior of cleaning at https://github.com/tlaurion/heads/tree/staging_cryptsetup_261 That was way too much and unexpected work.

[tlaurion]

@UndeadDevel: here is my log from bf2891c's test_reencrypt_ram.sh output automatically dropped under USB Thumb drive:

ram_reencrypt.log

Now at least we have logs to share to follow the rabbit down into the rabbit hole. Will get back to this and analyse later.

Direct excerpt showing how this is suboptimal:

# /tmp/disk8gb.raw is not a block device. Can't open in exclusive mode.
# Reusing open ro fd on device /tmp/disk8gb.raw
# Reusing open ro fd on device /tmp/disk8gb.raw
# Could not initialize userspace block cipher and kernel fallback is disabled.
# Failed to initialize userspace block cipher.
# Using temporary dmcrypt to access data.
# Allocating a free loop device (block size: 4096).
# Trying to open device /dev/loop0 without direct-io.
# Attached loop device block size is 4096 bytes.
# Calculated device size is 16293896 sectors (RW), offset 32768.
# DM-UUID is CRYPT-TEMP-temporary-cryptsetup-1215-0
# dm create temporary-cryptsetup-1215-0 CRYPT-TEMP-temporary-cryptsetup-1215-0 [ opencount flush ]   [16384] (*1)
# dm reload   (253:0) [ opencount flush securedata ]   [16384] (*1)
# dm resume temporary-cryptsetup-1215-0  [ opencount flush securedata ]   [16384] (*1)
# temporary-cryptsetup-1215-0: Stacking NODE_ADD (253,0) 0:0 0600
# temporary-cryptsetup-1215-0: Stacking NODE_READ_AHEAD 256 (flags=1)
# temporary-cryptsetup-1215-0: Processing NODE_ADD (253,0) 0:0 0600
# Created /dev/mapper/temporary-cryptsetup-1215-0
# temporary-cryptsetup-1215-0: Processing NODE_READ_AHEAD 256 (flags=1)
# temporary-cryptsetup-1215-0 (253:0): read ahead is 256
# temporary-cryptsetup-1215-0: retaining kernel read ahead of 256 (requested 256)
# Old cipher storage wrapper type: 2.
# Reusing open rw fd on device /tmp/disk8gb.raw
# Could not initialize userspace block cipher and kernel fallback is disabled.
# Failed to initialize userspace block cipher.
# Using temporary dmcrypt to access data.
# Calculated device size is 16293896 sectors (RW), offset 32768.
# DM-UUID is CRYPT-TEMP-temporary-cryptsetup-1215-1
# dm create temporary-cryptsetup-1215-1 CRYPT-TEMP-temporary-cryptsetup-1215-1 [ opencount flush ]   [16384] (*1)
# dm reload   (253:1) [ opencount flush securedata ]   [16384] (*1)
# dm resume temporary-cryptsetup-1215-1  [ opencount flush securedata ]   [16384] (*1)
# temporary-cryptsetup-1215-1: Stacking NODE_ADD (253,1) 0:0 0600
# temporary-cryptsetup-1215-1: Stacking NODE_READ_AHEAD 256 (flags=1)
# temporary-cryptsetup-1215-1: Processing NODE_ADD (253,1) 0:0 0600
# Created /dev/mapper/temporary-cryptsetup-1215-1
# temporary-cryptsetup-1215-1: Processing NODE_READ_AHEAD 256 (flags=1)
# temporary-cryptsetup-1215-1 (253:1): read ahead is 256
# temporary-cryptsetup-1215-1: retaining kernel read ahead of 256 (requested 256)
# New cipher storage wrapper type: 2
Resuming LUKS reencryption in forced offline mode.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Resuming LUKS2 reencryption.
# Progress 0, device_size 8342474752
# Calculating segments.
# Calculating hot segments (forward direction).
# Calculating post segments (forward direction).
# Setting 'hot' segments.
# Segment 0 assigned to digest 1.
# Segment 0 assigned to digest 0.
# Segment 1 assigned to digest 0.
# Segment 2 assigned to digest 0.
# Segment 3 assigned to digest 1.
# Reencrypting chunk starting at offset: 0, size :1073741824.
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# data_offset: 16777216
# Allocating buffer for storing resilience checksums.
# Checksums hotzone resilience.
# Going to store 8388608 bytes in reencrypt keyslot.
# Reencrypt keyslot 2 store.
# Reusing open rw fd on device /tmp/disk8gb.raw
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Device size 8359251968, offset 16777216.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Reusing open rw fd on device /tmp/disk8gb.raw
# Checksum:fb73f8bc2a92da33ea5e4f73f51afdec9bbb49d60bc69adebd2766f08a89258d (in-memory)
# Trying to write LUKS2 header (16384 bytes) at offset 16384.
# Reusing open rw fd on device /tmp/disk8gb.raw
# Checksum:d19a0c0ee6472baa03b6b04eee2b5103d3b7b07e42b86f6682622d5dec2a8758 (in-memory)
# Setting 'post' segments.
# Segment 0 assigned to digest 1.
# Segment 1 assigned to digest 0.
# Segment 2 assigned to digest 0.
# Segment 3 assigned to digest 1.
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Device size 8359251968, offset 16777216.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Reusing open rw fd on device /tmp/disk8gb.raw

[tlaurion]

Of course, kernel dependencies exploding and newer non-optimized libs/binaries break legacy boards compatibility, as seen in CircleIC checks here.

Will attack that later but same as before on block devices and same as for ram.raw disk in prior tests:

# Could not initialize userspace block cipher and kernel fallback is disabled.
# Failed to initialize userspace block cipher.

Even though

# Verifying key from keyslot 1, digest 1.
# Checking if cipher aes-xts-plain64 is usable (storage wrapper).
# Initializing reencryption (mode: reencrypt) in LUKS2 metadata.

Definitely missing something here. Will sleep on it.

[tlaurion]

Not sure what to do with the output of the block device reencryption log from 4cb56d4 block_reencrypt.log

[tlaurion]

@UndeadDevel had an insight on checking more closely resilience options. Under 2.7.0RC0 (unreleased: tarballs don't contain configure or other files we patch for cross-compiling, will need to wait anyway I do not really want RC0 of something) resilience speed can be augmented but for 2.6.1 --resilence=none brings us to ~normal speeds. To me this is around 110 MiB/s where my lower performance bar tolerance when reencryptpion was added under Heads, I twaked on multiple SSDs to substain 100 MiB/s under most of the accessible SSDs I found around, where chinese low end (Shame on you FANXIANG S101 provided by cheap refubishers that exist out there) were 37 MiB/s. I restate, no. We cannot afford having users reencrypt Heads flashed devices bought from 3rd party and have criticisms of that kind (Went though that already: no thanks).

Test satisfying under https://github.com/linuxboot/heads/pull/1541/commits/302452df8d1f010e5403648d584505e6f02fc044

TODO next:

• [ ] cleanup of kernel Kconfig options added vs master to reduce to bare minimal. Still to convinced adding suboptimal options even for older hardware which would benefit of xts-aes-plain64 anyway. Would add my veto here once again but would be willing to let Addendium or whatever its called if legacy boards still can build (yes. There are 3rd gen I3 out there. Once again, been there done that on x230 so not wanting to go there again.)

~The issue reported above is still a mystery though:~

~># Could not initialize userspace block cipher and kernel fallback is disabled.~ ~># Failed to initialize userspace block cipher.~

Edit: With --resilience=none, the issue of locking simply vanishes altogether and the errors about ciphers as well.

[UndeadDevel]

Just after your post 2.7.0-rc0 has been announced as released; the changelog is mostly about OPAL support, but the following bits also seem interesting for the issue at hand:

Support OpenSSL 3.2 Argon2 implementation.

  Argon2 is now available directly in OpenSSL, so the code no longer
  needs to use libargon implementation.
  Configure script should detect this automatically.

* Add support for Argon2 from libgcrypt
  (requires yet unreleased gcrypt 1.11).

  Argon2 has been available since version 1.10, but we need version 1.11,
  which will allow empty passwords.

* Link only libcrypto from OpenSSL.

  This reduces dependencies as other OpenSSL libraries are not needed.

Regarding resilience: it seems that in offline mode interrupting a --resilience=none reencryption is not necessarily unsafe, so if it really gives such huge performance boost (I'm surprised) then I guess it makes sense. This also further justifies modifying the warning message as I talked about in the issue thread (PR coming in few days hopefully).

[tlaurion]

Regarding resilience: it seems that in offline mode interrupting a --resilience=none reencryption is not necessarily unsafe, so if it really gives such huge performance boost (I'm surprised) then I guess it makes sense.

--resilience=none removes hotzone concept under reencryption. This means that there is no block checksuming nor resuming reencryption possible in case of power failure. That means that instead of locking/creating hotzone block at end of luks container, checksuming it and moving encrypted new block, cryptsetup does it inline but prevents interrupted reencryption from being resumable. This kinda doubles the speed.

Other notes from last commit:

--resilience=none : No hotzone created at end of container to write checksum value nor reopening of luks between block rewrite. No possibility of resuming interrupted reencrypt --disable-locks : skips aditional lock + check, assumes exclusive mode --force-offline-reencrypt : don't even try online mode --perf-no_read_workqueue : bypass async AIO read pooling (Thanks Cloudflare patchset) --perf-no_write_workqueue: bypass async AIO write pooling (Thanks Cloudflare patchset) --perf-submit_from_crypt_cpus : use single cpu crypt op to limit cache miss. Forces dmsetup compatible implementation

Weirdly enough, and that's ok again under Heads kernel, we do not have CONFIG_LIBAIO. No async buffer support under kernel. Still, cryptsetup requires libaio in newer version. Even weirder, we now make sure to not even use async IO now by forcing --perf-no_write_workqueue --perf-no_read_workqueue.

@UndeadDevel :

Just after your post 2.7.0-rc0 has been announced as released; the changelog is mostly about OPAL support, but the following bits also seem interesting for the issue at hand:

Support OpenSSL 3.2 Argon2 implementation.

  Argon2 is now available directly in OpenSSL, so the code no longer
  needs to use libargon implementation.
  Configure script should detect this automatically.

* Add support for Argon2 from libgcrypt
  (requires yet unreleased gcrypt 1.11).

  Argon2 has been available since version 1.10, but we need version 1.11,
  which will allow empty passwords.

* Link only libcrypto from OpenSSL.

  This reduces dependencies as other OpenSSL libraries are not needed.

Heads is a limitated beast on those regards.

• modules/openssl is not included nor compiled unless we have TPM2 on board for TPM2 toolstack requirements. So we cannot imply to have libcrypto
• Once again, we configure and compile cryptsetup to use the kernel crypto backend.
• We have libgcrypt, but as prior comment, we do not use it since kernel crypto backend is better at enforcing faster crypto backend taking adventage of cpu enhancements better (SSE2,AVX etc) for crytpo algos we use.
• We use internal Argon2 support.
• Opal support will be nice in the future. But I tend to get away of pushing for RC0 versions unless required. Also libgrcypt 1.11 is not released yet and going that direction would mean version bumping all gnupg toolstack.

I will post test results soon.

[tlaurion]

And yet again, reencrypting a real installation vs reencrypting a cryptsetup 2.6.1 luksFormat block device gives a lot of variation and requires more testing here again. Just stating --resilience=none shows "Setting hot segment" so not sure what is happening here. Doc says no checksuming, but it still seems that the region is written at end of LUKS container and then moved back in current reencrypted block instead of being in place? behavior different from https://github.com/linuxboot/heads/pull/1541#issuecomment-1831202928 where there is no

# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# data_offset: 16777216
# Allocating buffer for storing resilience checksums.
# Checksums hotzone resilience.
# Going to store 8388608 bytes in reencrypt keyslot.
# Reencrypt keyslot 2 store.
# Reusing open rw fd on device /dev/sda2
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# LUKS2 requirements detected:
# online-reencrypt-v2 - known
# Device size 248984371200, offset 16777216.

Anymore That was obtained from cryptsetup reencrypt /dev/sda2 --disable-locks --force-offline-reencrypt --debug --key-file /tmp/passphrase.txt. So --resilience=none changed speed from 50MiB/s there to 105MiB/s.

For example, in my tests right now, if I recreate container with luksFormat and then reencrypt it:

• reencrypt with --debug --resilience=none alone: Stable 116 MiB/s

If I reencrypt LUKS container created by stage 1 Q 4.2 RC5 installer:

• reencrypt with --debug --resilience=none alone: becoming stable at around ~105 MiB/s

Get home message: how the LUKS container was created changes the reencryption speed. And we need to make sure that the options we select optimizes for speed vs resilience. Heads is single user recovery environment and should permit users to reencrypt drive as part of OEM Rw-Ownership as fast as possible with current constraits.

Also noted that kernel config has not be reviewed here. Just injected Kconfig "best" practices but not for embedded environments.

[tlaurion]

Todo:

• [ ] add --perf* tweaks passed down to dmsetup to bypass kernel queues and write in a compliant way.

• [ ] Then retest without the dmsetup compliant way but still with perf tweaks for read/write buffer bypass.

• [ ] remove most kernel drivers added that aren't related.

Keep in mind that reencrypt tests need to be applied on top of an oo created and filled luks container otherwise reenncryption speed tests varies and are not meaningful to compare between them. Stay on same SSD drive and same platform to compare result

[tlaurion]

Poor SSD drive having to deal and exhaust all cells through wear prevention. This drive will have its livespan reduced by a lot will try to get health report later on.... Will post logs related to previous commits here, which test script was made to output results for specific commits. Also tests were made against q4.2 reinstall made before commit f08ea9a

As of now, it seems that all --perf* options are irrelevant as well as ram consomption, since switching to ramfs from tmpfs (tmpfs prevents ram to be filled above 8gb) has no impact at all, unless having reencrypted once q4.2 install is changing alignment or something, speed are steady at 109.x MiB/s with latest 92dcd67 which was not significantly better than 3847158 :

block_reencrypt_f08ea9ad7af49dbdd218362555634018649e7349.log block_reencrypt_fd4ac5c8a2accd5158429360b5f4da621ca4f45c.log block_reencrypt_50b18a5e25c96caaac317e74ab98cce0bb727a5b.log block_reencrypt_384715841bee04ea683a98ee4b54b36591f5ce9b.log block_reencrypt_92dcd67e331194c0081c304a1dd382f54523360e.log

find ./ -name "block_reencrypt_*.log" | while read filename; do echo "$filename"; grep Finished "$filename"; done
./block_reencrypt_50b18a5e25c96caaac317e74ab98cce0bb727a5b.log
Finished, time 36m25s,  231 GiB written, speed 108.6 MiB/s
./block_reencrypt_fd4ac5c8a2accd5158429360b5f4da621ca4f45c.log
./block_reencrypt_all_optimiz.log
Finished, time 36m14s,  231 GiB written, speed 109.2 MiB/s
./block_reencrypt_.log
./block_reencrypt_no_lock-offline-no_resilience.log
Finished, time 34m04s,  231 GiB written, speed 116.1 MiB/s
./block_reencrypt_384715841bee04ea683a98ee4b54b36591f5ce9b.log
Finished, time 36m14s,  231 GiB written, speed 109.2 MiB/s
./block_reencrypt_f08ea9ad7af49dbdd218362555634018649e7349.log
./block_reencrypt_92dcd67e331194c0081c304a1dd382f54523360e.log
Finished, time 36m10s,  231 GiB written, speed 109.4 MiB/s

[tlaurion]

okok... so dmsetup is used when online mode (if --force-offline-reencrypt is not specified). Will inspects logs more... Reinstalling q4.2 rc5 to use --force-offline-reencrypt as base

[tlaurion]

block_reencrypt_d313f448bd79e56193ed357ba75ad7400630f365.log

[tlaurion]

@UndeadDevel took me time to return to this, but retesting oem-factory-reset options on top of released Q4.2 release and now 4.2.1 release(to be done) made me aware that Heads was not able to reencrypt with previous options, being totally unaware of newer argon2 there and just failed there :/. Success at luksChangeKey though.

So I prioritized this over the weekend, revisited and cleaned commit trail and OP. Lost track of linux 5.10.10+ containing fixes for encryption speed from cloudfare merged upstream and retesting Q4.2 install now and will reapply reencryption from oem-factory-reset now default option.

Can you review the commit messages trail and current state of this PR to see if it matches the content of past discussions? I think it does, but landing your brain wouldn't hurt here. Ping me out please if you do.

[tlaurion]

Testing on real hardware postponed, only get hands on crappy chinese SSD that slowsdown after writing 4gb of data (fills SSD buffer at full speed) and then slows down to 32Mb/s.... Will backup stuff and reuse good SSD to test this one.

If anyone beats me to it, it should all be fine where Talos II cannot benefit from it because we use branched older 5.5-power kernel with patches not upstreamed to Linux. Again, we have no case of Talos II being sold by OEMs.

Important part is all resellers laptops/workstations known should permit reencryption on received shipment through re-ownership, as documented whys under the oem-factory-reset script itself.

[UndeadDevel]

I'm short on time these days, but I might be able to look it over this weekend or next week. Will @ you then.

[tlaurion]

Ok, finally got a hold of a Samsung 860 pro 250GB I repurposed, for which Q4.2.1 installed reencryption process on a x230 with current PR is currently progressing with speeds going up to a stable ~155 MiB/s (with some variation from 155.7 to 155.2) which final average speed calculated finalized at 155.4 MiB/s up to the end of the process in the lat 10%, which took less then initial ETA in an acceptable time of 26m04s. The initial reencryption acceptable speed calculates was of 140 MiB/s on a mx400 250 GB drive, where I don't recall the speeds for 860 drives, but the mental maximum accepted ration for users to find it acceptable to actually reencrypt was to be smaller then OS installation time, which was set around 30 minutes. We are still under those times on older hardware, where reencryption of 2TB drives on nv41 is hypothetically also around 30 minutes, but yet to be confirmed on real hardware and hands-on testing.

Never, ever use commodity China shipped Fanxiang S101 240G crap from china refurbishers (had tons of them which I replaced prior of sending to customers exactly for this). Those should be used in combination of a hammer for sole purpose of anger management purposes in case you need to destroy something with some positive violence. Please do invest in proper SSD drives for daily use, people. SSD drives like those just make computing a bad press for those sorts of things.... SSD memory buffers being filled by intense IO on the same disk being read and write is a disaster for such cheap drives, which cannot withstand anything else then limited random access and definitely not a reencryption use case. Reencrypting such crappy SSD drives will limit throughput to a crawling speed (20MiB/s?!?!?!) when the SSD's RAM is filled. Seems like there is 2GB or ram onboard and operations in firmware are expected to be async only and really low throughput. Anyway.

I also think I might have got input for https://github.com/Nitrokey/heads/issues/48, which I will redo once again, but seems linked to usb drivers not being loaded when the dongle is actually booted?! If the device is already plugged in without drivers being loaded, when comes the time for the OEM to probe for device capabilities, the OEM gives the user a chance to connect it, but reconnecting it won't do it unless its connected to a different port?!

Meaning the OEM factory reset should ask to remove the dongle, load drivers and then ask the user to insert the dongle once usb drivers are loaded. To be confirmed on next test run with DEBUG activated to try to get what is happening or to be retested on qemu.

[tlaurion]

Next test session with pending changes turning AIO on in kernel and turning on again writing queue bypass (I know two changes=bad) reduced the speed to a stable 150 on fresh Q4.2.1 install.

Note that a regression was noticed which is quite important. luksChangeKey op reuses the same keyslot for op. All good. cryptsetup 2.6.1 when doing reencryption rotates the key into next avail keyslot and wipes previous one after op. Oups.

This means that the current logic for DUK and DRK reencryption needs to change. I have some thoughts, but that would require refactoring DUK (cannot expect it to be slot 1 while DRK is 0 anymore), with code taking care of previous assumptions to migrate DUK in slot 8(last) if under slot 1, permitting DRK to be in whatever slot it was found matching passphrase and wiping the other found if two keyslots were found, otherwise proposing to the end user to wipe any other slots not being DRK prior of resealing DUK...

Any thoughts on that welcome, otherwise I will try to implement changes soon when the idea matures enough in my head.

[tlaurion]

Ok, retesting on clean Q4.2.1 without Async (AIO) in kernel but with write queue bypass in alone added in test case.

Speeds are still increasing, up to 157.5 MiB/s at 15%. Average still increasing now, so the bypass of queues alone as an improvement on slower and older hardware, which shows those have a positive impact for reencryption. 157.7 MiB/s at 80% and 100%! As stable ETA and reencryption speed as one can get it seems. 25m41s to reencrypt 237Gb on a 250GB drive on a x230. Not bad really.

Thanks to cloudfare for improving the situation here, upstreaming and collaborating with whole toolstack changes and documenting this properly. I think we finally have maximum reencryption performance possible taking in consideration current best practices.

Last commits changed the logic of kexec-seal-key to not expect DRK to be under key-slot 0 (reencryption rotates key-slot to next avail), while dealing with previous imlmentation to warn user if we are about to wipe a keyslot which is not 1 or 8 prior of wiping all other keyslots not matching DRK passphrase, which permitted to unlock the LUKS container and ask the user to wipe any keyslot not being 1 or 8, wiping all others automatically. This seems safe enough.

Will be able to test the logic after this round of reencryption by changing the logic on luks-functions to not bind any of the luksChangeKey or reencryption to any specific slot as before, which would now be irrelevant.

@UndeadDevel the code under kexec-seal-key is adding a luks key. Therefore, we could enforce better defaults there, if desired. Let's see what is being done by default from cryptetup 2.6.1 here. As stated before, I would prefer staying away and relaying on what cryptsetup enforces as the best defaults there. When comes to DUK, lets remember that the DUK "passphrase" here, more a key itself really, is a 128 chars randomly generated (with now HW backed randomization if available) and therefore I strill believe this to be irrelevant since that keyslot cannot even be filled with weak "passphrase". Nobody will type 128 chars here and no collision is theoritically possible, even?

[tlaurion]

Hmmm.

a second reencryption on the same original Q4.2.1 install now reencrypts at 75.9 MiB/s, and the HDD led is mostly not showing any activity?!

EDIT: my bad.... --resilience=none --force-offline-reencrypt --disable-locks got wiped.

[tlaurion]

Still some commits cleanup to do but this is ready for a first review

[tlaurion]

Some raw notes though, not sure if with all of those tests I kills cells, but time was of 26m54s at 150.6 MiB/s to write 237GiB, which is a regression on past test. Not a lot but still. All worked as expected, no regression.

[tlaurion]

OEM factory reset without accepting default, accepting DRK passphrase change + reencryption, setting one shared passphrase for all provisioned secret and then setting a boot default on reboot with TPM DUK worked as expected.

And booting boot default with TPM DUK key booted into Q4.2.1 with said DUK key passed.

[tlaurion]

@UndeadDevel really sorry about the delay fixing this, back onto it and I think i'm nearly done fixing things and proposing changes. Will push changes here soon when tests are completed against qemu under nix against luksv1/luksv2 container. I was nearly done but had to context switch and now getting back from context switch (which takes a while to get back to where I was unfortunately)

I saw #1678, reviewed quick (thanks and sorry you had to do this, we deduplicated brain power), I think everything is under current PR as of now but might have missed something. Will push code changes soon, staging under https://github.com/linuxboot/heads/compare/master...tlaurion:cryptsetup_version_bump-reencryption_cleanup-staging2?expand=1

[tlaurion]

@UndeadDevel works on luks v1 and v2 but commit trail needs cleanup and the whole code needs my own review

Edit: works only on single luks drive setup. Not yet done.

[UndeadDevel]

Thanks, @tlaurion ; ~~had a quick glance (not a review) at the changes and one thing I noticed is that the code asserts "7" to be the last LUKS1 slot, when it's actually "8" (as per the LUKS1 spec...LUKS1 key slots start at 1 and are 8 in total, while LUKS2 key slots start at 0 and are 32 in total - thereby 31 is the last LUKS2 key slot as correctly noted in your updated code). Edit: nevermind, I just tested this with a LUKS1 file volume created by cryptsetup 2.6.1 and at least in that configuration LUKS1 key slots start at 0, not 1, so the spec is confusing in this regard and your code correctly identifies key slots 7 and 31 as the last slots for LUKS1 and LUKS2, respectively.~~

Another issue is that you seem to have included one of my commits from #1646 twice, thus in effect fixing the wrong PIN (Admin PIN, not User PIN)...I did make a bit of a mess with that PR, so mea culpa. I think if you revert this commit it should be fixed properly.

[tlaurion]

Not yet done. It doesn't play nicely over qubes 4.2.1 installed with BTRFS (two luks containers) on which I quickly tested this morning, causing regression in handing multiple luks containers and also promoting twice for drk which should be one and failing if unlocking other containers with dark fail, as well as wiping the key slots for all luks devices, not just the first encountered one.

Another round of commits will happen before cleaning up.

@UndeadDevel thanks for the heads up.

[tlaurion]

Everything functional. Cleanup tomorrow. Damn things are big sometimes.

Just simulated Q4.2.1 2xLUKSv2 containers (BTRFS deployment) under Qemu: works.

Will test on real hardware but should be the same exact thing: LUKS passphrase expected to unlock multiple LUKS devices configured for setup; if one fails user is guided to change LUKS container passphrase (which exists independently through Options menu).

All seems good.

[tlaurion]

Ok. Will try to clean up the commit trail now since boards build (this PR bumps Linux kernel version which will break other UNMAINTAINED boards a little bit more, which is what happens when things are untested promptly as needed).

[tlaurion]

Nope. Not today. Priorities changed once again

[tlaurion]

luks reencryption still requires a loop to do all detected luks containers (otherwise user needs to do other container from menu) on btrfs q4.2.1 alternative disk partition scheme.

Weird that cryptsetup accepts /dev/sda2 /dev/sda3 on command line but only does the first, normal I guess.

[tlaurion]

Also missing, when renewing DUK, host needs to reboot, otherwise typing TPM DUK results in PCR unsealing errors (normal, since we just loaded usb drivers on non-hotp boards, which aren't part of expected TPM DUK unseal op.

Todo:

• [ ] add loop in oem-fatory-reset or luks renncryption function to make sure that 2 luks partition scheme setups are reencrypted/passphrase changed with same passphrase
• [ ] force reboot after DUK renewal in TOTP/HOTP reset/resealing path to not throw scary users to the user for non-hotp/usb keyboard enabled configurations which don't expect usb drivers to be loaded when unsealing TPM DUK.