Drop support for non-generated build modes

[poettering]

mkosi currently has to different ways to build things:

1. regular builds — where we first set up a disk image, create an fs in it, mount it, and install linux into
2. "generated" builds — where we set up the OS in the workdir first, and finally pack it up into a disk images

Originally all builds were regular builds, but eventually more and more "generated" build schemes where added, to cover for tarballs, squashfs, "minimized build", /usr-only builds...

I'd like to propose that we drop support for regular builds entirely, and just build everything the same way: prepartion in some dir in the workdir, and then packing things up via a tool that builds a disk image from it (which can be a simple "cp" if the used image format has no tool for packing things up)

Why? To bring the combinatory explosion down a bit, and simplify things. More systematic behaviour is a good thing.

What would we lose this way? I kinda liked the fact that the file system features we provide while building the image ultimately matched exactly what we'd generate, which is great, in particular for btrfs images and such. Things might become a bit slower in some cases, as we'd always need a final copy operation.

incremental builds wozuld have to changed a bit. i.e. incremental build snapshots would have to be taken on an fs tree level, we could no longer do this on the loopback file level.

THoughts?

[behrmann]

I like the idea, when I first tried to factor the distros out into their own modules, I stumbled over disentangling that from the outputs, so I think this might go hand in hand.

[DaanDeMeyer]

For me, it would depend on how much slower incremental becomes. I quite like that I can build and boot into a qemu vm with the latest systemd changes in 12s or so when using btrfs with cached images and an incremental systemd build on my crappy laptop. Of course, like you say, the combinatorial explosion we have now isn't ideal. It's next to impossible to make sure everything works correctly.

What about doing the opposite? Building the generated stuff in a disk image as well instead of a separate workdir and just packing the disk image contents into a tar, squashfs, w.e. at the end?

[poettering]

What about doing the opposite? Building the generated stuff in a disk image as well instead of a separate workdir and just packing the disk image contents into a tar, squashfs, w.e. at the end?

Hmm, interesting idea. Also has the nice effect of putting a natural quota on things, which I think is a good idea.

Maybe this is indeed the way to go. i.e. always start out with a loopback image, consider it our "storage pool" that might or might not end up being the image in the end.

So what about this approach: instead of starting it out with fully partitioned block device which we then only use partly and then later on patch, do it like this:

1. for each partition we need (i.e. ESP, root partition, /home, /srv, …) generate a separate loopback file, with the actual fs to use in it.
2. Then build the image from that
3. One of these two:
   1. Finally, combine the resulting file systems into a combined disk image, and add a partition table around it. Maybe use systemd-repart for this part, it should have all the right options already. Use reflinks when combining (after all, reflinks can be about partial files, too) so that the dropping in of the partitions is practically free on btrfs+xfs (i.e. file systems where reflinks exist)
   2. Alternatively, pack it up in a tar ball, mksquashfs, …

This reduces special handling, since we won't have distinct codepaths for generated and non-generated images anymore, except for that fact we pack things together differently as last step.

And performance on reflink-capable file systems should be ideal, since we can take the incremental snapshots in perfect time, and the glueing together at the end should be perfectly quick too. And the incremental stuff would be naturally available for all outputs, without special handling.

makes sense?

[lucasdemarchi]

would we loose -t directory then?

[poettering]

hmm, nah, we shouldn't lose that. Maybe a "cp -av" out of the prepared image and into the dir is OK though?

ah, fuck, maybe we cannot really get rid of non-generated builds after all. Sniff.

[DaanDeMeyer]

Why wouldn't cp -av suffice? @poettering's suggestion seems ideal if it works. -t directory might become a bit slower but I can live with that.

[lucasdemarchi]

I think it would more natural to prepare the workspace in a directory and later create the image and copy things over. But I also understand the motivation for doing the other way around. I think both can be made to work and it would be nice to remove the extra complexity we have now.

[DaanDeMeyer]

This was implemented with the migration to systemd-repart, so let's close the issue