nixos-generators

nixos-generators copied to clipboard

see: https://discourse.nixos.org/t/virtualization-ova-ovf-esxi-support-extended-ova-customization/7536

When trying to provision a VM on VMware using a .ova image generated with nixos-generators, I get a Unsupported hardware family 'virtualbox-2.2'. error.

$ cat flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nixos-generators = {
      url = "github:nix-community/nixos-generators";
        inputs.nixpkgs.follows = "nixpkgs";
    };
  };
  outputs = { self, nixpkgs, nixos-generators, ... }: {
    packages.x86_64-linux = {
      virtualbox = nixos-generators.nixosGenerate {
        pkgs = nixpkgs.legacyPackages.x86_64-linux;
        modules = [ ./minimal.nix ];
        format = "virtualbox";
      };
    };
  };
}

$ cat minimal.nix
{ config, lib, pkgs, ... }:
{
  environment.etc."ssh/auth_principals/root".text = "admins";
  environment.etc."ssh/trusted-user-ca-keys.pem".source = ./trusted-user-ca-keys.pem;

  services.openssh = {
    enable = true;
    permitRootLogin = "no";
    passwordAuthentication = false;
    extraConfig = ''
      AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
      TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
    '';
  };

  users.users.root = {
    initialPassword = "nixos";
    openssh.authorizedKeys.keyFiles = [ ./authorized_keys.txt ];
  };
}

On a related note: this project could really use some documentation and usage examples.

Apparently, it can be worked around with

 pkgs.runCommand "fix-esxi-image" { }
  ''
    ova=${vboxImage}/*.ova
    mkdir $out
    ${pkgs.cot}/bin/cot edit-hardware $ova -v ${vmx} -o $out/nixos.ova
  '';

but I have no idea, where that is supposed to go.

I actually have the same requirement, and think I got this working but haven't had access to esxi to test it. If my solution works I'll try and post it when I'm back at my laptop tonight.

Ultimately, the way to support this is probably to add an additional format but I haven't looked into how to do that yet

@mayl if you tell me what the solution is, I'd be happy to try it on an ESXi. Adding an additional format for taht should be as easy as copying https://github.com/nix-community/nixos-generators/blob/master/formats/virtualbox.nix and overwriting the settings there.

Yea, we'd need to add a format file but I think we'd need to do more than just add an existing imports and I'm not totally sure how that plays out. Again, haven't actually looked into it too closely.

Saw on discourse that it looks like you figured out more or less where I landed. Let me know how your vmx testing goes, I still haven't been able to test against a real ESXi instance.

hmm, the nixos-generators format for vmware is just a very thin wrapper around: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/vmware-image.nix so ideally the stuff which is broken should be fixed there? But new formats like esxi could be helpful, sadly I don't know anything about that

Actually, around https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/virtualbox-image.nix, because vmware-image.nix just produces a vmdk that doesn't work with most ESXis, so no device and machine information, just the disk, while virutal-box.nix makes an .ova that at least gets you one step closer to an actual provisioned machine.

I've found out a couple of settings that work, each time after, finding something else that ESXis complain about. Ran out of time before my christmas vacation. Maybe I'll go on after it, but I'm far from an expert, too. Hope someone more skilled will take pity.

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/virtualization-ova-ovf-esxi-support-extended-ova-customization/7536/6

Okay, so over a year later, here is what I ended up using. It's far from perfect but it works for me:

  packages."x86_64-linux" = {
    # Note: Exclusively works with specific versions of both the unstable
    # (19cf008b) and stable channels (b83e7f5) and nixos-generators 
    # (30516cb2). Only dog knows why.
    # The sole way to test it, is to actually deploy the resulting image
    # using ovftool (if you want to use the tool). Deploying it over webui
    # is a bit less picky, so you might get away with other commits.
    nixovabase = let
        pkgs = nixos.legacyPackages.x86_64-linux;
        unfixed = nixos-generators.nixosGenerate {
          pkgs = pkgs;
          modules = [ ./machines/ovabase.nix ];
          format = "virtualbox";
        };
        vmx = "vmx-13";  # see: https://kb.vmware.com/s/article/1003746
      in pkgs.runCommand "nixovabase" { } ''
        ova=${unfixed}/*.ova
        mkdir $out
        # cp $ova "$out/unfixed.ova"  # debug
        ${pkgs.cot}/bin/cot --force --verbose edit-product $ova -p 'Some Info' -o nixos.ova
        ${pkgs.cot}/bin/cot --force --verbose edit-hardware nixos.ova -v ${vmx}
        tar xf nixos.ova
        sed -i -E 's/^(\s*<(ovf:)?ProductSection)>\s*$/\1 ovf:required="false">/' *.ovf
        sed -i -E "s/^(SHA1\(nixos.ovf\)=\s*).*$/\1$(sha1sum nixos.ovf | cut -d ' ' -f 1)/" *.mf
        ${nixunstable.legacyPackages.x86_64-linux.ovftool}/bin/ovftool --lax --sourceType=OVF --targetType=OVA nixos.ovf $out/nixos.ova
        # tar cf $out/nixos.ova *.ovf *.mf *.vmdk
      '';

wehere ovabase is:

{ config, lib, pkgs, ... }:
{
 imports = [ ];

 # ... probably you want user configuration and openssh as a minimum here ...

 # System configuration
 networking.hostName = config.system.nixos.label;  # env NIXOS_LABEL nix build --impure ...
 virtualbox = {
   # see: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/virtualbox-image.nix
   memorySize = 4000;  # MiB
   params = {
     # audiocontroller = "off";
     audio = "none";
     audioout = "off";
   };
 };
 virtualisation.vmware.guest.enable = true;
}