Add support for A/B root partitions

[c4t3l]

Hello,

I have a need for A/B root partition scheme for some remote edge and iot device deployments. I currently deploy a mkosi generated image to baremetal host then create a "root partition b" with the same size etc as the original root partition. I can hack mkosi and force creation of a second root partition, but I think the solution needs to have support from maybe another mkosi.* file or additional flags from cli. Is this functionality on anyone's radar?

[behrmann]

Interesting use case. Out of curiosity, how do you determine which root partition is booted?

[c4t3l]

Systemd will automount whichever root partition with a GUID of 4f68bce3-e8cd-4db1-96e7-fbcaf984b709 (x86_64) without GPT attribute "GUID:63" active. This information is provided in the Discoverable Partitions Specification ( https://www.freedesktop.org/wiki/Specifications/DiscoverablePartitionsSpec/). It is a simple matter of toggling the GUID:63 attribute on your root partitions. I have tested this manually on several prototype systems (mostly Dell Edge 3000 and Intel Nuc) and it works very well. I have attempted to add the additional partition to the appropriate function in mkosi (determine_partitions), but it seems to fail at actually creating the filesystem. It combines both root partitions into one large one.... sigh... Perhaps I will have more time to hack on it over the coming holidays.

At any rate, I feel this use case has tremendous benefit over traditional package update methods. In particular, I have several thousand of these edge devices that are on very low-bandwidth networks. We have run into situations where updating applications stall due to packet drops, rpm database corruption etc. This method would allow me to use modern tools like casync to download reboot into a running tested environment.

On Wed, Nov 27, 2019 at 7:16 AM Jörg Behrmann [email protected] wrote:

Interesting use case. Out of curiosity, how do you determine which root partition is booted?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/systemd/mkosi/issues/379?email_source=notifications&email_token=AB4HRQYRREFJNL3JX3KWO2LQVZXKDA5CNFSM4JKACX5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFJO3JQ#issuecomment-559082918, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4HRQZS6U7WBMWM2QEDAYLQVZXKDANCNFSM4JKACX5A .

[behrmann]

Cool, I've never noticed that the discoverable partition spec has that toggle. You don't have any method though to automatically fall back to a working root partition if a reboot fails, right? So the use case right now is only to push a new image and after successfully doing that to toggle the partition and reboot?

Could the reason for both your root partitions being combined lie with sfdisk, maybe it's combining duplicate lines? I guess the solution would then be to create the second root partition with a dummy GUID and changing it afterwards.

[poettering]

BTW, my plan was to add a small tool to systemd called "systemd-repart" or so, which could do minor repartitioning during boot. Specifically, enlarge partitions and add in partitions, i.e. everything that is purely "additive". This tool would then be configured via drop-in files that allow definitions of partitions, their size constraints, as well as the number of instances they should exist in. Then we'd automatically run that tool during boot if any drop-in files exist. So the idea was then that mkosi would always generate minimal images, but on first boot systemd enlarge / maybe, add in a second partition for / and add in /var + /home + /var/tmp + swap or whatever else you need.

In fact in preparation of that we already added the x-systemd.growfs and x-systemd.mkfs mount options, which would be used to setup/resize the fs inside the partitions after they are created.

Long story short: i think doing A/B stuff is fantastic and the way to go, but I think it would be nicer to d do that at boot time, not at image generation time. I mean, GPT partition tables pretty much cater for atomically updating them, so we could do all this in a way that is safe towards abnormal shutdown if we do it right.

Why is doing that at boot time better than at image generation time? Smaller images to download, and automatic adjustment to the actual media size of the hw.

[NicolasT]

(Sorry if this is off-topic, just wondering)

Using real GPT partitions for A/B root selection could be cumbersome because partition sizes are somewhat fixed (unless one provisions them with some margin), right? As such, if v1 is, say, 10GB in size, v2 is 11GB and v3 is 14GB, and all you have is a 25GB disk, you better plan things (upfront, somehow!) accordingly.

So, A/B booting got me wondering: what if instead of using partition(s) for the root FSs, why not drop these as files in a (large) FS (maybe using FS or block-level dedupe?), then losetup the desired one at boot-time and use the corresponding loop device (which could be GPT-partitioned again) as the root device?

[DaanDeMeyer]

As @poettering mentioned, this should be delegated to systemd-repart, so let's close the issue here. To deal with the issue @NicolasT mentioned, we could use dm-linear to grow partitions as needed without having to plan things up front.