How to modify /boot firmware to allow BTRFS root?

[n8henrie]

Hello,

Thanks for a great walkthrough. I can build an SD image with Docker!

Unfortunately, I've been working fruitlessly on getting a Pi to use a BTRFS root for a while.

I was hoping I could use this builder to create a boot partition whose u-boot images had BTRFS support enabled; unfortunately, adding:

boot.initrd.supportedFilesystems = [ "btrfs" ];
boot.supportedFilesystems = [ "btrfs" ];

to my sd-image.nix produces an identical boot partition as compared to not having this (diff -r shows no differences, md5sum on *.bin is identical).

Do you happen to know why the /boot files aren't being affected?

[Luflosi]

I was able to get this to work on a Raspberry Pi 4 by simply copying the generated disk image to the SD card, then resizing the second partition with fdisk, overwriting it with mkfs.btrfs then mounting it. I also created a loop device with losetup -f -P /path/to/image.img, mounted the second partition and copied everything over. The configuration you edit before building the disk image needs to already have the correct configuration (/boot with the correct label and the correct btrfs sub volumes if you created any). Don't try to also put /boot on Btrfs, it won't work. I didn't have to modify the boot partition in any way (I did edit the label with fatlabel though). There is also no need for boot.initrd.supportedFilesystems or boot.supportedFilesystems since that is taken care of automatically by specifying fsType = "btrfs".

[Robertof]

Sorry for the late response all - unfortunately I have no idea on how to achieve something like that as I am not a btrfs user, but thanks @Luflosi for contributing! I'll leave the issue open in case anyone has anything else to report - I'd also suggest to reach out to one of the NixOS communities to see if anyone has experience with this!

[n8henrie]

Thanks for your responses!

@Luflosi -- I've been working on this for a couple days now. I tried a configuration with the below and a few other subvols, but it's not working yet. Also, it seems like it should be possible for the sdImage to be created ready-to-go, without having to manually mkfs.btrfs, mount a loop device, copy files over, etc -- that's what I'm ultimately hoping to accomplish! But yes, just getting it working at all would be a great first step.

Also, on an RPi3, I did have to use fileSystems = lib.mkForce, though I can't remember now what error I was getting without that.

I did edit the label with fatlabel though

What did you mean by this?

Trying again now with omitting discard=async (I can't recall how new the kernel is on 21.05 for an RPi3):

  fileSystems = lib.mkForce {
    "/boot" = {
      device = "/dev/disk/by-label/FIRMWARE";
      fsType = "vfat";
    };
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "btrfs";
      options = [ "noatime" "ssd_spread" "discard=async" "compress-force=zstd" "autodefrag" "subvol=@" ];
    };
    "/var" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "btrfs";
      options = [ "noatime" "ssd_spread" "discard=async" "compress-force=zstd" "autodefrag" "subvol=@var" ];
    };
...

@Robertof no worries, thanks for leaving this open!

I'd also suggest to reach out to one of the NixOS communities to see if anyone has experience with this!

I've tried a few avenues with little success so far:

• https://discourse.nixos.org/t/raspberry-pi-nixos-on-btrfs-root/
• https://github.com/NixOS/nixpkgs/issues/133435
• https://www.reddit.com/r/NixOS/comments/o6sjbu/rpi_nixos_on_btrfs/

¯\(ツ)/¯

[Luflosi]

Yes, having an image ready to go would be nice but I don't know how to do that.

I did edit the label with fatlabel though

What did you mean by this?

I did sudo fatlabel /dev/sdx1 BOOT but as long as you have the same label in the NixOS configuration (FIRMWARE in your case, which is the default I think), it shouldn't matter.

I created the Btrfs filesystem with sudo mkfs.btrfs --metadata=single --label sd --features no-holes /dev/sdx2. The --label part is important. You should probably replace sd with NIXOS_SD.

What error message do you get when attempting to boot the Pi?

[n8henrie]

Yes, having an image ready to go would be nice but I don't know how to do that.

That's my goal, hopefully will make some headway with the help of this project. Still don't know much about nix obviously, but learning every day.

The --label part is important.

I bet that's it! I'll get back soon.

[n8henrie]

Nope, that wasn't it (at least not by itself -- I haven't tried without discard=async yet).

What error message do you get when attempting to boot the Pi?

This repeats a few times:

missing environment variable: bootfile
Retrieving file: pxelinux.cfg/default-arm-bcm283x
...Waiting for Ethernet connection... done.
*** ERROR: `serverip' not set

Eventually:

Cannot autoload with TFTPGET
U-Boot>

There is an unrecognized filesystem type in there somewhere, but it scrolls by pretty fast (and unfortunately Ctrl-S and Scroll Lock aren't working for some reason).

[n8henrie]

FWIW, here is a little script I whipped up to automate modifying the image / creating my subvols of preference / mounting with my btrfs flags of preference / rsyncing over the data from the ext4 partition.

NB: For future readers, I am no bash pro, so this could be dangerous. Please read through it and make sure you double check my work.

https://gist.github.com/n8henrie/6fbef60f1f9c4afc4d8f952bf27ac631

[Luflosi]

I think you should replace ${device}1 with ${device}p1 and ${device}2 with ${device}p2 in your script. But that's not the problem here. I'm not familiar with the NixOS boot process on Raspberry Pis but try

boot.loader.generic-extlinux-compatible.enable = false;
boot.loader.raspberryPi.enable = true;
boot.loader.raspberryPi.version = 3;

in your configuration.

[n8henrie]

I think you should replace ${device}1 with ${device}p1 and ${device}2 with ${device}p2 in your script.

I don't think so -- this is referring to a block device, where I have my SD card mounted for burning (not another loop device). It seems to be working as is.

$ ls -l /dev/sdf*
brw-rw---- 1 root disk 8, 80 Oct  7 05:15 /dev/sdf
brw-rw---- 1 root disk 8, 81 Oct  7 05:15 /dev/sdf1
brw-rw---- 1 root disk 8, 82 Oct  7 05:15 /dev/sdf2

I'm not familiar with the NixOS boot process on Raspberry Pis but try

I'll give it a try, but the native Raspberry Pi bootloader doesn't have BTRFS enabled either; I have several Pis running BTRFS, and any distro that uses the Pi bootloader requires a separate initrams and config changes to tell the kernel to load it.

[Luflosi]

I don't think so

You're right.

the native Raspberry Pi bootloader doesn't have BTRFS enabled either

That's why the /boot partition is not on Btrfs.

requires a separate initrams and config changes to tell the kernel to load it

On my Pi, NixOS automatically copies the kernel and initramfs to /boot.

[n8henrie]

That's why the /boot partition is not on Btrfs.

I guess this is what I was referring to: https://github.com/raspberrypi/linux/issues/1761

I'm still trying to sort this out. Was getting a bunch of errors about conflicting values with boot.loader.generic-extlinux-compatible.enable, so I tried lib.mkOverride and lib.mkForce without luck, now getting errors about boot.loader.generic-extlinux-compatible.populateCmd' is used but not defined.