impermanence

impermanence copied to clipboard

I just switched my root to be on tmpfs. I decided to start with mounting the real root fs to /persistent and bind-mounting everything from there. Except that it wouldn't boot because it tried to mount /etc before mounting /persistent. I discovered that naming the mount point /apersistent fixed it tho x) Apparently the filesystems are loaded alphabetically. Is that expected or did I miss something? Will this stay that way with future nix versions? If that's expected behavior then I think it deserves to be mentioned in the README.

Yes, you need to mark the persistent filesystem neededForBoot in order for it to be mounted early enough. This is mentioned in the README.org at the end of the NixOS section (on line 61).

I did that, but it was not enough, because /nix is also neededForBoot, but /nix is supposed to be bind-mounted from /persistent. What happens is that during stage-1 the initrd knows it needs to mount /nix, /persistent, and others, but doesn't know that /persistent must be mounted first. So it tries to mount /nix (which is alphabetically the first), and fails.

In the past I think I worked around this by making my persistent fs be /nix and make /nix/state my impermanence dir. You can see the setup here: https://github.com/lovesegfault/nix-config/blob/master/systems/fourier.nix

It's not great, but it works.

I see. Is there a reason to set it up like this, if you don't mind me asking? If you want to use /nix for persistent storage, you can just use a subpath of it, like /nix/persistent.

I also use /persistent for persistent storage, but it's a mounted btrfs subvolume, not a bind mount. Here is my filesystem config: https://github.com/talyz/nixos-config/blob/master/machines/evals/configuration.nix#L82-L110. I also create a fresh btrfs subvolume on boot and use it as root, which saves memory and provides a way to recover in case of crashes, power outages or forgetfulness ;)

My main reason for setting up /persistent the way I do is that I started from an existing normal nixos installation. So /persistent points to the root of a normal installation, with its /nix, /home, /etc and so on. Btw I want to mention that apart from having to name the mountpoint /apersistent, the rest of the transition was extremely smooth! Fresh btrfs sounds like a pretty cool idea! But so far my non-persisted / takes 20M tops.

Okay, that makes sense. I guess the clean solution would be moving to something a bit more flexible, like btrfs subvolumes or separate partitions for persistent state. I don't think we can do much here - if the bind mount dependency ordering should be applied anywhere, it's in NixOS.

Yeah, most of the time my root is pretty small too, but when I used tmpfs I ran out of memory more than once due to not planning ahead 😅

Yeah, we'd need NixOS to support specifying dependencies between mounts or something like that. That's why I thought mentioning it in README is the best we can do from here

It is the only thing we can do, but I'm not sure how to mention it or if we should. This really is a last resort setup which should be avoided if it can be; I don't want to suggest that it's somehow common, recommended or even something most people will have to bother with. Do you, or anyone else, have a suggestion? Also, it might be good to open an issue against the NixOS/nixpkgs repo, so this might get fixed and we have something to refer to.

I'm running into this issue now as well: https://github.com/ryantm/agenix/issues/45#issuecomment-847852593

If I try to set "/etc/ssh" as neededForBoot it hangs the boot process because of the ordering issue.

Update

Seems there is a PR to address this, I'm gonna test it out: NixOS/nixpkgs#86967

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/384

I haven't tested but it should be fixed by https://github.com/nix-community/impermanence/pull/115