LXC: Changes don't persist on restart

[ncryptid]

Whenever I make a change to an LXC container, either by manually editing /etc/nixos/configuration.nix or push changes from a config management tool like morph or nixops, the changes to the container don't persist if it gets restarted.

I generate an image from this config:

{ config, lib, pkgs, ... }:

{
  boot.isContainer = true;
}

I then change /etc/nixos/configuration.nix on the container to reflect the boot options and add a test:

{ config, pkgs, ... }:

{
  imports = [  ];
  boot.isContainer = true;
  environment.systemPackages = with pkgs; [ hello ];
  
}

After rebuilding the system, hello is found and I can run it. If I print the current generation I'm on...

[root@nixos:~]# hello
Hello, world!

[root@nixos:~]# nix-env --list-generations --profile /nix/var/nix/profiles/system
   1   2020-11-21 09:55:44   
   2   2020-11-21 10:03:18   
   3   2020-11-21 10:04:49   
   4   2020-11-21 10:12:04   
   5   2020-11-21 10:12:27   
   6   2020-11-21 10:30:44   (current)

... and then restart the container, not only will it revert back to the base config used to generate the image (and therefore hello will be gone), but it also creates another generation from the base config for some reason:

[root@nixos:~]# nix-env --list-generations --profile /nix/var/nix/profiles/system
   1   2020-11-21 09:55:44   
   2   2020-11-21 10:03:18   
   3   2020-11-21 10:04:49   
   4   2020-11-21 10:12:04   
   5   2020-11-21 10:12:27   
   6   2020-11-21 10:30:44   
   7   2020-11-21 10:32:46   (current)

Not sure if this is expected behavior or I'm not configuring something correctly, but I don't have this issue with containers for other OS images.

[Zer0-]

+1 i am also having this issue

[Lassulus]

Oh well, I'm not actually sure if this is indeed expected behaviour. I will try to reanimate my lxc build environment and try to investigate. But to be honest, there is a lot on my plate and I'm not sure when this will happen. So I encourage you to investigate yourself, since nixos-generators is a very minimal wrapper around existing nixos/nixpkgs infrastructure.

anyways, thanks for your feedback and sorry I can't be more helpful

[Zer0-]

The problem for me is I am not sure what nixos's boot process is like to read the current derivation and apply it. Especially if it's in a container. I have tried naively putting "nixos-rebuild test" in postBootCommands but that didn't work. I have also tried adding the postBootCommands from here without success: https://git.dsg.is/david/nixpkgs/-/blob/dc9374427357a942a27e36fc8cc0674aa8cbc22c/nixos/modules/profiles/docker-container.nix

[ncryptid]

I gave up on this about a month ago because I also could not figure out what in NixOS's boot process determines which derivation to use but I suppose I'll have to go back and debug some more. This definitely seems like it's not specific to nixos-generators but if I figure out what the problem is I'll post here because at the moment this bug kind of defeats the purpose of using lxc.

[ghost]

1. Add this to your system config:

  imports = [
    <nixpkgs/nixos/modules/virtualisation/lxc-container.nix>
  ];

2. nixos-rebuild switch
3. Profit

Explanation:

LXC "boots" the container by starting /sbin/init inside the container. In a NixOS container this is actually a generated bash script, which among other things contains the system configuration that will be activated at "boot". It is added to the tarball by nixos-generate, but it is not updated when you build a new generation, unless it is configured to do so. lxc-container.nix imports nixos/modules/profiles/docker-container.nix, which adds adds a activationScripts.installInitScript.

Be aware that this also includes other things like nixos/modules/profiles/minimal.nix, which sets environment.noXlibs to mkDefault true. If your container suddenly wants to rebuild half the world, try settings environment.noXlibs back to false.

[towards-a-new-leftypol]

@petabyteboy What you said makes sense, but doesn't work when I tried this unfortunately.

I added the line to my imports, ran nixos-rebuild switch then stopped the container by running "poweroff" as root inside the container. Then I used "lxc start my-container" to boot it up again, and I could not ssh into it because the ssh service was not started.

I had to revert to my old trick of running lxc exec my-container -- /run/current-system/sw/bin/bash and running nixos-rebuild test inside of it to restore the state and start the systemd units. It seems there is as step missing.

[n0emis]

@towards-a-new-leftypol You may need to change lxc.init.cmd to /init for this to work...

[towards-a-new-leftypol]

SWEEET this fixes a long-standing annoyance, now everything works on boot!

@n0emis instead of changing config on the host away from default I put this in my configuration.nix:

  # Install new init script
  system.activationScripts.installInitScript = lib.mkForce ''
    mkdir -p /sbin
    ln -fs $systemConfig/init /sbin/init
  '';

[towards-a-new-leftypol]

Thank you @petabyteboy , @n0emis !

[aanderse]

SWEEET this fixes a long-standing annoyance, now everything works on boot!

@n0emis instead of changing config on the host away from default I put this in my configuration.nix:

  # Install new init script
  system.activationScripts.installInitScript = lib.mkForce ''
    mkdir -p /sbin
    ln -fs $systemConfig/init /sbin/init
  '';

I believe this is the default now. I think this issue should be closed as fixed.