lx: NixOS Support

[cmacrae]

Preface

Hello everyone 👋

I'm opening this issue as a sort of report for "Hey! We're here and we'd like to see this happen".
Supporting NixOS as an official lx-brand distro would be very nice to have.
I won't blabber on about all the cool things you could achieve with it, as that's readily apparent to anyone familiar with both illumos (SmartOS/Triton in particular) and NixOS - but I dream of using a combination of the metadata API and NixOS expressions to achieve truly reproducible, declarative service images!

Past efforts

As it turns out, it has been achieved in the past. I believe @vizanto, with a bit of help from @edolstra, during NixCon 2015 managed to produce a bootable image for use on SmartOS. It's available here:
http://datasets.at/ui/#!/configure/22ba59c6-8c49-11e5-87e5-7fbffbc05623

As far as I understand it, after a little chatter with @vizanto on Twitter, this was achieved within an lx-alpine zone. I don't have the full details, as unfortunately it wasn't captured/documented anywhere - but I believe this was done by extracting Nix within the zone and doing a sort of in-place replacement, then removing any remaining Alpine resources. Probably not far from the method that nixos-in-place uses.

Current points of reference

Right now, as far as a working example of this, all we have to go on is the above linked image, and this hardware-configuration.nix that @vizanto kindly uploaded. Though, I'm guessing from the shell output at the bottom, that there are issues with it.

I imported the current image on datasets.at into my private Triton deployment. When trying to provision an lx zone with it, it failed due to some Triton nodejs script failing - something to do with networking, I can't recall exactly at the moment, but this is something I'll get when I'm back home and have a minute.

Desirable traits/outcomes of this issue

For me, personally, what I'd like to see as an outcome of this issue boils down to two points:

• (Eventual) Official support for running NixOS on SmartOS (namely, on Triton).
• A solid, reproducible pipeline for creating NixOS base images (I'll touch on possibilities for this below).

Recent efforts

I spent a bit of time over the past weekend trying to hack something together, before I stumbled across @vizanto's image/references in GH issues.
My efforts were based around two approaches, one of which I feel could potentially be the right route to take:

• Using nixos-install to bootstrap an install into a directory, which could then be tar'd up to form the root filesystem tarball for the base-image.
• Using lib/make-system-tarball.nix/lib/make-system-tarball.sh to the same affect.

Using either of the above methods means we could use Nix expressions to declare properties of the base image during the build pipeline.
Both nixos-install and lib/make-system-tarball.nix provide a means of expressing with Nix what the resulting tarball will contain.
To this affect, we could essentially achieve everything the current 'lx-build' scripts for other distros do, but in a solid reproducible manner.

I feel nixos-install is likely going to be the best route for this. When inside a NixOS environment, we can use a collection of Nix expressions that nixos-install will evaluate at the time of execution.

Wheras with lib/make-system-tarball.nix, as far as I understand it, the approach would be more like:

• Build Nix expressions in the current Nix environment
• Graft the resulting stores into the tarball using the contents parameter

Which I imagine there would be problmes with, as the NixOS environment you're sat in will likely not resemble the target environment in terms of system/hardware configuration(s).

One current problem is that the lx_boot script has some logic to determine the target OS based on filesystem resources. The image on datasets.at has a dummy /etc/alpine-release file to circumvent this. I also attempted this, before knowing about the image, but must've messed it up, as it didn't work for me 😅

Two main points from me concerning lx_boot and zone booting in general on SmartOS:

1. How does one figure out what steps are necessary to take in the distro specific lx_boot scripts?
2. What filesystem resources are expected to be present in an image to be able to boot?

In light of the above, what's the best method of setting up a development environment to get started on all this stuff? Of course, I can't go in editing the lx_boot script, and any other system resources necessary, in my Triton deployment.
Are there instructions anywhere to detail how I can achieve (loosely) the following?

• Make changes to the sources in this repository
• Build a resulting SmartOS image
• Boot said image (VirtualBox or something, I guess?)
• Start testing changes by attempting the main goals outlined in this issue

I suppose those last two points are kinda moot, but just for clarity's sake; I'll leave them there