impermanence icon indicating copy to clipboard operation
impermanence copied to clipboard

How do I know when a file or directory will be mounted during the boot process?

Open PhDyellow opened this issue 6 months ago • 4 comments

I am moving my configuration to impermanence, using btrfs subvolumes to erase root on each boot. The persistent directory is on another subvolume in the same btrfs partition, and /nix is inside the persisted subvol

I have hit a number of issues with programs not having the needed files, and did a bit of digging into the system logs.

Given the following impermanence config:

environment.persistence."/persistent" = {
	  enable = true;
	  hideMounts = true;
	  directories = [
		{
		  directory = "/secrets"; 
		  mode = "0700";
		  user = "root";
		  group = "root";
		}
		"/etc/ssh"
		"/etc/secureboot"
		"/var/lib/nixos" 
		"/var/lib/systemd"
		"/nix"
		"/var/log"
	  ];
	  files = [
		"/etc/machine-id"
	  ];
	};


home.persistence."/persistent/home/phil" = {
  directories = [
	".ssh"
	".emacs.d"
        "syncthing"
	}
  ];
  files = [
	".local/share/fish/fish_history"
  ];
  allowOther = false;
};

The systemd logs show that /nix, /var/lib/nixos and /var/log are mounted very early in the boot sequence, near the end of stage-1-init.

Then /etc/machine-id is bound after just stage-2-init, while systemd is running but before journalling is started.

Finally, /etc/secureboot, /var/lib/systemd, /secrets and /etc/ssh are mounted after a number of other services have started up, including local-fs-pre.target.

The home manager files are mounted even later, and a number of services have started by the time they are mounted.

Programs that were caught out were syncthing, which started before the syncthing config directory had been mounted in my home directory, and sshd, which did start after /etc/ssh was mounted, but because NixOS had already created and configured /etc/ssh, the nixos-configured sshd_config file was hidden by the bind mount.

I was able to fix syncthing by modifying it's systemd service with the following nixos module:

impermanence-syncthing = {config, lib, pkgs, ...}:
  {
	systemd.services.syncthing.after =  [
	  "multi-user.target" 
	];
	# Default dependencies cause syncthing-init.service to
	# set "Before=multi-user.target" which, along with setting
	# "After=multi-user.target" for syncthing.service, leads to
	# circular dependency.
	# Turning off default dependencies, explicit dependencies were
	# sufficient.
	systemd.services.syncthing-init.unitConfig.DefaultDependencies = false;
  };

I find the order directories are mounted during the boot process confusing and unexpected. I want all the persistent directories and files bound or symlinked before system services start trying to read config files. I expected NixOS to write sshd_config into /etc/ssh beside the persisted machine keys.

I think this could be resolved with a good understanding of when persistent entries are mounted. Could someone explain the logic to me?

PhDyellow avatar Aug 23 '24 02:08 PhDyellow