lx: NixOS Support

[cmacrae]

Preface

Hello everyone 👋

I'm opening this issue as a sort of report for "Hey! We're here and we'd like to see this happen".
Supporting NixOS as an official lx-brand distro would be very nice to have.
I won't blabber on about all the cool things you could achieve with it, as that's readily apparent to anyone familiar with both illumos (SmartOS/Triton in particular) and NixOS - but I dream of using a combination of the metadata API and NixOS expressions to achieve truly reproducible, declarative service images!

Past efforts

As it turns out, it has been achieved in the past. I believe @vizanto, with a bit of help from @edolstra, during NixCon 2015 managed to produce a bootable image for use on SmartOS. It's available here:
http://datasets.at/ui/#!/configure/22ba59c6-8c49-11e5-87e5-7fbffbc05623

As far as I understand it, after a little chatter with @vizanto on Twitter, this was achieved within an lx-alpine zone. I don't have the full details, as unfortunately it wasn't captured/documented anywhere - but I believe this was done by extracting Nix within the zone and doing a sort of in-place replacement, then removing any remaining Alpine resources. Probably not far from the method that nixos-in-place uses.

Current points of reference

Right now, as far as a working example of this, all we have to go on is the above linked image, and this hardware-configuration.nix that @vizanto kindly uploaded. Though, I'm guessing from the shell output at the bottom, that there are issues with it.

I imported the current image on datasets.at into my private Triton deployment. When trying to provision an lx zone with it, it failed due to some Triton nodejs script failing - something to do with networking, I can't recall exactly at the moment, but this is something I'll get when I'm back home and have a minute.

Desirable traits/outcomes of this issue

For me, personally, what I'd like to see as an outcome of this issue boils down to two points:

• (Eventual) Official support for running NixOS on SmartOS (namely, on Triton).
• A solid, reproducible pipeline for creating NixOS base images (I'll touch on possibilities for this below).

Recent efforts

I spent a bit of time over the past weekend trying to hack something together, before I stumbled across @vizanto's image/references in GH issues.
My efforts were based around two approaches, one of which I feel could potentially be the right route to take:

• Using nixos-install to bootstrap an install into a directory, which could then be tar'd up to form the root filesystem tarball for the base-image.
• Using lib/make-system-tarball.nix/lib/make-system-tarball.sh to the same affect.

Using either of the above methods means we could use Nix expressions to declare properties of the base image during the build pipeline.
Both nixos-install and lib/make-system-tarball.nix provide a means of expressing with Nix what the resulting tarball will contain.
To this affect, we could essentially achieve everything the current 'lx-build' scripts for other distros do, but in a solid reproducible manner.

I feel nixos-install is likely going to be the best route for this. When inside a NixOS environment, we can use a collection of Nix expressions that nixos-install will evaluate at the time of execution.

Wheras with lib/make-system-tarball.nix, as far as I understand it, the approach would be more like:

• Build Nix expressions in the current Nix environment
• Graft the resulting stores into the tarball using the contents parameter

Which I imagine there would be problmes with, as the NixOS environment you're sat in will likely not resemble the target environment in terms of system/hardware configuration(s).

One current problem is that the lx_boot script has some logic to determine the target OS based on filesystem resources. The image on datasets.at has a dummy /etc/alpine-release file to circumvent this. I also attempted this, before knowing about the image, but must've messed it up, as it didn't work for me 😅

Two main points from me concerning lx_boot and zone booting in general on SmartOS:

1. How does one figure out what steps are necessary to take in the distro specific lx_boot scripts?
2. What filesystem resources are expected to be present in an image to be able to boot?

In light of the above, what's the best method of setting up a development environment to get started on all this stuff? Of course, I can't go in editing the lx_boot script, and any other system resources necessary, in my Triton deployment.
Are there instructions anywhere to detail how I can achieve (loosely) the following?

• Make changes to the sources in this repository
• Build a resulting SmartOS image
• Boot said image (VirtualBox or something, I guess?)
• Start testing changes by attempting the main goals outlined in this issue

I suppose those last two points are kinda moot, but just for clarity's sake; I'll leave them there

[cmacrae]

To touch on my final point of building a SmartOS bootable image; I see that there are instructions here:
https://github.com/joyent/smartos-live

However I'd like to understand how I could do so using modified illumos-joyent code for testing changes.

[nshalman]

https://wiki.smartos.org/display/DOC/Building+SmartOS+on+SmartOS

[Ericson2314]

@cmacrae Wow. Totally coincidentally, I was just today looking into supporting cross compilation to illumos in Nixpkgs, a long-standing personal goal that sat at the back of my mind while I overhauled the nixpkgs cross compilation infra. Let me know if you would also be interested in working on that, now that the generic infrastructure is in place.

[cmacrae]

@Ericson2314 Oh, cool! I'm not sure how much I'd be able to help other than having illumos systems to test on and under heavy guidance 😅

[Ericson2314]

@cmacrae Heavy guidance is fine. I must admit I have somewhat limited time, but so far it's worked well where I've advised a number of people working on specific cross use-cases, having myself spearheaded the general infrastructure. But I'd be willing to try help out more with this, at least until we could assemble a good bit more people.

[Ericson2314]

A good first step for you would be trying to add the Illumos linker to nixpkgs (to be built with regular Linux stdenv and packages). Does that sound doable to you? I can then work on teaching bintools-wrapper about it (a more involved task that overlaps greatly with the Linux <-> Darwin cross stuff I'll probably be doing for work).

[bahamat]

@cmacrae:

I think this would be great, but we may not currently have the resources to maintain a new image family. And for security reasons, we can't allow externally contributed origin images to be loaded into Joyent's cloud.

I also think it's a good candidate for an RFD to solicit feedback from the wider community.

Going further, I think we'll need to go at this in stages.

1. Creating NixOS KVM instances
2. Creating the lx images
3. Adding necessary NixOS support to lx_boot

KVM images are created from a machine-image repo. You can search Joyent's org for repose named mi-*. There's also an example repo. You'll want to start off by cloning this and creating a kvm image.

LX images are created using a KVM instance of the same type. E.g., CentOS LX is made using CentOS KVM. So once you have a KVM image working you can start on generating lx images. The repos for that are named *-lx-brand-image-builder. However, we don't have an example repo for LX yet. You'll want to create a repo that follows a similar pattern.

And eventually, you'll need to update lx_boot. Building SmartOS is easy following the guide that @nshalman pointed you to. In order to do a build using a custom repo/fork, edit the configure.smartos file. The format is fairly straightforward. Once you think you have it working, you'll need to follow the processes outlined for SmartOS contributions and the illumos Developer's Guide.

[plitc]

@cmacrae

1. run your own dsapid (for publishing images) https://github.com/MerlinDMC/dsapid
2. pick up the old community driven nixos-lx image (and develop it) http://dsapid.root1.ass.de/ui/#!/configure/22ba59c6-8c49-11e5-87e5-7fbffbc05623

[karaiwulf]

Hey folks, I've been looking into this recently. @bahamat, do the instructions you've posted still hold?

[bahamat]

@karaiwulf Currently we're using a two-stage process. First, lx-images to create the installation payload and then smartos-lx-img-builder to create the image.

It's a bit under-documented at the moment, but hopefully shouldn't be too hard to figure out by following the existing distros.

[dcarosone]

Just stumbled across this issue, and would be keen to see it happen!

[karaiwulf]

@dcarosone I dunno, the build outlined above explicitly relies upon Docker, and the NixOS community appears to be averse to dockerizing their OS, which is understandable. If you can find a decent source for a general purpose NixOS docker container, it would be relatively trivial to get it done. The closest thing I could find was nixery.dev, but testing proved those images are not particularly general purpose, and the process of dockerizing NixOS doesn't appear to be terribly straight forward either.