Add serpentOS and moss support

[hunger]

SerpentOS (https://serpentos.com/) is a new Linux distribution that is currently investing heavily into new tooling to build and package software.

It also uses a new package manager called moss to quickly install software.

This commit adds support for both serpentos as well as its package manager.

There are a couple of limitations though:

• I did not add tests as building moss needs to be build from rust sources at this time
• mkosi can not create disk images from inside serpent OS (systemd-repart is too old there)
• You can not create bootable SOS images (systemd-ukify is missing)

The devs are focusing on the tooling right now and did not update the packages in a while.

This PR potentially enables doing tests using mkosi definitions, so I think this PR is valuable even at this time. I think the bit of code that this PR adds will also stay stable: None of the commands used changed in a long time (they were even stable across a rewrite of the entire packager in rust).

[hunger]

I don't think these integration test failures are due to my change, it seems to happen for other actions as well.

[DaanDeMeyer]

Yeah don't worry about those I'm fighting them at the moment

[DaanDeMeyer]

So playing around with this, first thing that seems to be missing is /bin/sh. Which package provides that in serpent os? I can install bash but it doesn't provide /bin/sh. This is important for mkosi's scripts as they tend to use /bin/sh in their shebang.

[hunger]

@DaanDeMeyer: So playing around with this, first thing that seems to be missing is /bin/sh. Which package provides that in serpent os? I can install bash but it doesn't provide /bin/sh. This is important for mkosi's scripts as they tend to use /bin/sh in their shebang.

The dashpackage contains /usr/bin/sh.

[hunger]

@DaanDeMeyer: moss should really have an option to look for repositories and configuration in general in a directory outside of the root directory. All the other package managers we support have this and it allows us to keep the image itself clean of package manager specific details.

moss has some unique properties I have not seen in other package managers. It has a content-addressed file store in .moss, which also contains all the repo data, available packages, etc. On package installation it fills the file store in /.moss and only afterwards "blits" the package contents in the form of hard-links to the data in the file storage into /usr(it does not package anything outside /usr).

At a later point you can also ask moss to "blit" a set of packages into other places: That creates entirely separate file trees (consisting of hard-links to the file storage in .moss).

So in the ideal setup I would create a folder in some global cache somewhere and use that for all moss images that have the same repositories set up (e.g. by doing a SHA of the repo config or something). I would set that as root directory when running moss -- for all mkosi images I want to create for that configuration of repositories.

Once all that is done, I can then ask the installation in that cache/moss-root-directory to blit only the packages the individual mkosi image needs into that mkosi's root directory. That will create hard-links when possible but will gracefully fall back to copying the files over if hard-links do not work.

That would be a very efficient cache for reposiroty state, packages, etc. with all the settings being outside the mkosi image and would significantly speed up image creation, too.

Do you think I can do that within mkosi? It is a bit different from how other package managers behave :-)

I'm also missing an option to specify a cache directory for moss to use so we can have moss use mkosi's cache directory.

There is no such option. With the setup described above that might not even be necessary.

I'm also curious what package signing looks like on Serpent OS. Is any signing done at all? If so, where is the keyring located that moss should use?

They are working towards package signing, but it is just not there yet. The entire thing is very early.

[hunger]

I did leave .moss in the root directory: Otherwise any image with an installed moss binary will not work.

[hunger]

mkosi -d serpentos -f qemu starts systemd in the image, but fails at some later point.

SOS comes with a ready-made initrd in the kernel package, so I needed to handle that case in mkosi/__init__. Mkosi still seems to package up modules anyway, even though that should not be necessary at all.

[DaanDeMeyer]

@hunger We should not use the SerpentOS initrd when building mkosi images. Our current policy is that if no initrd is provided by the user we build one ourselves. So you'd have to add definitions for SerpentOS to mkosi/resources/mkosi-initrd so we can build SerpentOS initrds.

[hunger]

@DaanDeMeyer: OK, but then mkosi will behave significantly different to "normal" SOS during early boot as those all will use the same initrd as built (and signed -- once they guys get to that).

Anyway: I'll take a look and see what I can do later tonight.

[DaanDeMeyer]

@hunger From what I read on their blog, they weren't even planning on using systemd-boot or UKIs though, so mkosi will produce something significantly different from the official image either way. At some point we'll add support for picking up signed prebuilt UKIs likely, though I don't really see the point in picking up prebuilt initrds.

[hunger]

@DaanDeMeyer: Minimal changes left outside the moss.py and serpentos.[py|conf] files now. The initrd is handled entirely by mkosi now.

[ikeycode]

Hi @DaanDeMeyer - Ikey here, Serpent OS lead. I've just been made aware of this PR, so I thought I'd answer some of your queries.

• moss-rs doesn't yet take non rootfs configurations but it is planned (it uses the same usr/etc layering approach of systemd). It operates on the contextual root directory (-D)
• package signing will be introduced in a v2 bump using multisig (See https://github.com/serpent-os/moss-rs/issues/50)
• We plan to use systemd-boot (+shim) - we ship prebuilt initrds and the dracut package. Our linux-kvm package shows a simple example of creating an initrd, it's just not something we'd support in an installed system.
• For now, we force /bin/sh via dash until an alternatives system is implemented. Note also we enforce usr-merge, so: moss install 'binary(sh)' will yield the correct candidate.
• The team and I have mentioned to @hunger we do have our own plans for container builds, and can't ourselves throw support into this effort right now (volatility of the system internals)
• We do currently issue nightly builds of moss-rs for testing, if its useful we could work out some signed releases on a specific branch.

Like said, while we're not formally supporting this effort we're not going to block progress or innovation. Any questions , don't hesitate.

Edit: Trigger queries

Currently work is underway to run triggers within a namespace, operating on a temporary view of the new transaction root before "activating it". ie new root in staging tree as /usr, and run triggers defined in YAML under clone based namespace. This is due to land soon, so for now manual fudgery is likely required re first boot.

[DaanDeMeyer]

@ikeycode

moss-rs doesn't yet take non rootfs configurations but it is planned (it uses the same usr/etc layering approach of systemd). It operates on the contextual root directory (-D)

Where does it look for its configuration though? In the rootfs or outside of it?

We do currently issue nightly builds of moss-rs for testing, if its useful we could work out some signed releases on a specific branch.

We can just build it from source probably, no big deal

Like said, while we're not formally supporting this effort we're not going to block progress or innovation. Any questions , don't hesitate.

Yeah we don't really care about being anything official, as long as the package manager makes it easy enough and the whole thing is relatively stable, we can probably support it.

[ikeycode]

@DaanDeMeyer right now it looks within the rootfs, though the plan is to support all required configs via CLI/clap. moss repo add creates files in /etc under -D root.

[hunger]

With these changes mkosi -d serpentos -f qemu boots into a serpentos system up to the command prompt. That's as good as it gets right now.

Actually configuring the initrd-creation did wonders;-) Thanks for pointing me to it, I completely missed that config section.

I'll probably need to look into the generated initrd next and see whether there are unnecessary files in there now (there probably are;-).

[hunger]

The mkosi initrd is 52MB, the serpentos initrd is 15MB, I definitely need to do something to trim that down. I'll leave that for next weekend though (I doubt I'll find the time during the week, sorry).

[hunger]

@DaanDeMeyer: Do you consider the question about the install method answered now? I did move the setup of /boot there, but there really is nothing else to do. moss will always create every folder necessary but for /boot, whatever I threw at it.

[hunger]

Also added a patch that slims down the initrd a bit. Still way too big, compared to the "real" initrd SOS builds.

[Winterhuman]

Here's some suggestions for slimming things down further (bare in mind I haven't seen the initrd myself, so I'm kinda guessing here):

• /usr/share/licenses (it's not just populated by the license package, assuming that's a package under moss).
• /usr/lib/gconv (honestly suspicious of this one, not noticed any problems so far)
• Assuming un-split gcc-libs package, see the Arch Linux mkosi-initrd config.
  • It also includes some /usr/lib/udev, /usr/lib/systemd/catalog, and other lines that may be relevant regardless.
• If the krb5 package is included, consider removing its /usr/bin/* commands.
• /usr/share/terminfo (maybe keeping /usr/share/terminfo/l/linux for VTs).
• If moss's kernel has built-in consolefonts, /usr/share/kbd/{consolefonts,consoletrans,unimaps} can be removed.
  • Also, most of the /usr/share/kbd/keymaps & /usr/share/X11/xkb contents too (though they're a bit harder to handle).

[hunger]

@winterhuman: I used the arch config for inspiration. What is in there is already getting cleaned from the serpentos initrd. Much of it is just not there, so the list is shorter.

I hope nobody wants to convert character sets in the initrd, as I do clean that out now :-)

[hunger]

I do not think the opensuse failures in CI are related to my changes. I did touch the opensuse mkinitrd config (moving udev from the common into distro-specific config), but thta should have an effect later in the build process I think.

[hunger]

@DaanDeMeyer Sorry for letting this linger for so long, I think I'll get around to do something about this in the train to Brussels on Friday. I'll try to chat you up after one of your presentations:-)

[DaanDeMeyer]

@hunger You might want to take a look at https://github.com/systemd/mkosi/pull/2336 already as well as that will drastically change things which you'll have to adapt this PR to.

[hunger]

@DaanDeMeyer: Thanks for the pointer :-)

I am a bit unsure how to do support this cache at this time. moss is rather bare-bones still and I have not even seen where it caches downloaded packages. Not sure it even caches downloaded packages at all (but I have not really looked;-). It has its content-addressed storage space in .moss and "blits" /usr from contents in there (using hardlinks or copies if hardlinks are not an option). So there is little reason to keep downloaded packages after their contents got added to the storage repository. (Think: ostree and maybe a bit of nix/guix splashed on top).

I am wondering whether I can just put the entire .moss folder in the cache... But then what do I do when moss gets installed into that image? It needs the /.moss in that case... . How would I copy that over into the image is needed?

I am also not 100% sure it will be able to do hard-links when bind-mounting .moss from some cache location who-knows-where into the image's /.moss to populate /usr -- which would drasticvally improve space requirements when moss is installed and I need to ship /.moss in the image.

[keszybz]

• For now, we force /bin/sh via dash until an alternatives system is implemented. Note also we enforce usr-merge, so: moss install 'binary(sh)' will yield the correct candidate.

If this was already discussed above in one of the collapsed comment, apologies. Github doesn't make it easy to search the whole conversation.

Debian tried to switch to dash-as-/bin/sh, and it was a lengthy and painful failure. There are bazillions of scripts which just use #!/bin/sh, but actually depend on bash features or run incorrectly with dash. So implicitly using dash is a constant fight. On the other hand, dash may be a bit smaller and faster, but the differences are insignificant, and any non-toy installation will require bash anyway, so the highest possible benefits are tiny. This is just a waste of time in the long run.

[hunger]

Thanks for the review @keszybz!

I need to rebase to mkosi master (quite a few changes in there) and then do some extensive testing before this can get merged.

SerpentOS has gotten out of its "we focus on tools" phase and started packaging a lot during the last weeks. So this PR can (hopefully) get away with less changes to the global package lists now.

[ikeycode]

Debian tried to switch to dash-as-/bin/sh, and it was a lengthy and painful failure. There are bazillions of scripts which just use #!/bin/sh, but actually depend on bash features or run incorrectly with dash. So implicitly using dash is a constant fight. On the other hand, dash may be a bit smaller and faster, but the differences are insignificant, and any non-toy installation will require bash anyway, so the highest possible benefits are tiny. This is just a waste of time in the long run.

I'm inclined to agree fwiw. Sure, CONF_SHELL can be overriden, and tbh we never pursued it for POSIX it was entirely to speed up package builds. It does do that, but it is an awful pain in the arse. Even using dash for the recipe scripts makes life harder, ie brace globs, single vs double square bracket, etc.

I've largely been waiting for someone to verbalise it better than myself as an excuse to nuke it from orbit. The maintenance story around dash isn't fantastic, and I'm aware of a few situations where it segfaults.

[Conan-Kudo]

Please write this up somewhere so that I can take it back to the openSUSE guys who recently decided busybox-sh should be /bin/sh sometimes. 😦

[bluca]

• For now, we force /bin/sh via dash until an alternatives system is implemented. Note also we enforce usr-merge, so: moss install 'binary(sh)' will yield the correct candidate.

If this was already discussed above in one of the collapsed comment, apologies. Github doesn't make it easy to search the whole conversation.

Debian tried to switch to dash-as-/bin/sh, and it was a lengthy and painful failure. There are bazillions of scripts which just use #!/bin/sh, but actually depend on bash features or run incorrectly with dash. So implicitly using dash is a constant fight. On the other hand, dash may be a bit smaller and faster, but the differences are insignificant, and any non-toy installation will require bash anyway, so the highest possible benefits are tiny. This is just a waste of time in the long run.

Yeah unless you are absolutely strapped for resources and counting milliseconds, don't bother with dash. Also I recommend staying away from 'alternatives' systems - they go behind the back of packages and make a mess of things, and don't really work for read-only rootfs images.

[DaanDeMeyer]

@ikeycode Any chance you could put your cache/state directory somewhere in /var (/var/cache or /var/lib) as well? Currently moss is the odd one out by using a new top level directory instead of the standard FHS ones.