FS corruption after hibernation

[xzfc]

trafficstars

The BTRFS script provided in the README.org has some problems that could result in filesytem corruption or data loss.

1. Filesystem corruption. This script runs both during the normal boot and when resuming from hibernation. Mounting BTRFS right before resuming is equivalent to mounting the same filesystem twice at the same time (without proper safeguards). I had my filesystem corrupted and turned into an unmountable/unbootable state because of this.

   The solution: instead of postDeviceCommands, use postResumeCommands. See stage-1-init.sh source for the difference.

   Also, see "How to fix the FS corruption after hibernation" below.

2. Wiping the wrong subvolumes. The function delete_subvolume_recursively might delete all the subvolumes on a disk (not just a particular subdirectory of /old_roots). This can occur if old_roots and its children are plain directories rather than subvolumes, which can happen after restoring disk contents from a backup.

   The reason is that the -o option in btrfs subvolume list prints the subvolumes of the given parent subvolume, not the subvolumes of the given directory. To illustrate:

   $ ls
   nix  old_roots  persistent  root
   
   $ sudo btrfs subvolume list -o .        # note no subvolumes in `old_roots`
   ID 259 gen 2575 top level 5 path root
   ID 260 gen 2334 top level 5 path nix
   ID 261 gen 2575 top level 5 path persistent
   
   $ sudo btrfs subvolume list -o ./root        # root is a subvolume, the output is expected
   ID 262 gen 1862 top level 259 path root/srv
   ID 263 gen 1865 top level 259 path root/var/lib/portables
   ID 264 gen 1865 top level 259 path root/var/lib/machines
   ID 265 gen 2511 top level 259 path root/tmp
   ID 266 gen 2511 top level 259 path root/var/tmp
   
   $ sudo btrfs subvolume list -o ./old_roots/2024-11-20_05:35:10      # (!) not a subvolume, the output lists unrelated subvolumes
   ID 259 gen 2575 top level 5 path root
   ID 260 gen 2334 top level 5 path nix
   ID 261 gen 2575 top level 5 path persistent
   

   In this case, delete_subvolume_recursively /old_roots/2024-11-20_05:35:10 will happily delete /root, /nix, and /persistent.

   Possible solutions:

   • Delete subvolumes using the --recursive flag instead of awkwardly unreliably bash function. Tho this flag is added very recently (v6.12, 2024-11-29) and not available in NixOS 24.11 which has v6.11.
   • Use the plain rm -rf. Starting from the kernel v4.17, rm could be used to delete subvolumes (see BTRFS changelog). Might be slower than btrfs subvolume delete.
   • Don't rely on the -o flag and filter subvolumes on our side using awk or bash string manipulation.

   Combining the first two solutions:

   btrfs subvolume delete --recursive -- "$i" || rm -rf -- "$i"
   

3. find -mtime +30 filters based on a boot time, not the "last used" time. Therefore, if you reboot after an uptime of more than one month, you'll end up with an empty /old_roots. Also, the same if your system clock got reset (e.g., CMOS battery is dead). Solutions:

   • Keep at least one old root regardless of its age.
   • Delete old roots based on their date compared to the date of the latest root, not the current date.

4. (not a corruption/data loss, but a slight inconvenience) I'm not sure why the date format is +%Y-%m-%-d_%H:%M:%S with %-d and not with %d. This makes lexically sorting unusable, e.g.: 2025-01-1, 2025-01-20, 2025-01-3.

How to fix the FS corruption after hibernation

In my case, the situation was similar to this SO thread, with these errors in dmesg (citing from SO):

parent transid verify failed on 109973766144 wanted 1823 found 1821

What did help:

1. Mount BTRFS in ro mode, copy the most important data to another disk and then recreate the filesystem from scratch. If you are lucky, it might work until you stumble upon a corrupted file. If not, see the next step.
2. btrfs restore to copy the data, then recreate FS from scratch.

What did not help: much everything else.

After copying the data back, make sure that /old_roots contains only subvolumes, otherwise the script will wipe your drive (see "Wiping the wrong subvolumes" above).

[talyz]

1. I see. Personally, I don't use hibernation, so this did not occur to me. I'm also using a systemd intitrd version of this, which I now realize was never added to the README. Anyway, I've changed the README to use postResumeCommands now.
2. While the error condition you list is a case of user error, it would still be nice to resolve. The first solution is the obvious choice, since I only wrote the function because there was no native recursive subvolume delete. We'll have to wait for it to be widely available, but I guess that shouldn't be too long. I tried using using rm -r first, but it's not reliable enough, sadly.
3. (and 4) Yeah, while in theory this seemed nice, it simply doesn't work well enough in practice. I think the better solution is to label them serially and just keep a set number of roots instead.

[accelbread]

First issue is actually exactly why I added postResumeCommands to nixpkgs; nice to see it put in use.

[ElvishJerricco]

FYI, to prevent the same problem in systemd initrd, you need your service to have after = [ "local-fs-pre.target" ];

[workflow]

FYI, to prevent the same problem in systemd initrd, you need your service to have after = [ "local-fs-pre.target" ];

@ElvishJerricco FYI the folllowing seems to work to bypass running on resume entirely.

(only relevant under systemd initrd)