Recovery image and squashfs

[davidcassany]

This issue is a follow up of https://github.com/rancher-sandbox/cOS-toolkit/issues/1054#issuecomment-1047862704.

Currently the recovery image can be in two flavors:

1. Equal (bit a bit) to the active partition (understand it as a bootable backup of the firstboot system)
2. A devoted recovery type image (understand it as a minimalist system that can be used for administration purposes: restore, upgrade, debug, etc. )

For the first, the recovery image is literally a copy of the original active image deployed during the installation, hence it is a non compressed image (no squashfs involved).

For the second, the recovery image is a plain squashfs filesystem containing the root tree. Why squashfs? two reasons:

• it can be created without needing privileges (this was supposed to be very important at a time)
• a compressed filesystem is way smaller, it definitely reduces the footprint on disk at a reasonable performance cost (IO performance is not a big deal for recovery)

To me this divergence on the use of squashfs or not is annoying and causes some strange logic in elemental procedures (upgrade, install, reset, etc.). Specially because, in general, the strategy to identify the system we are running at moment is by checking the current root filesystem label, the problem is that squashfs does not support labels (nor UUID either). This results into having to run some non obvious checks. Also the deployment of an image differs if we are in the first or second use case and that causes complex dual paths on upgrade, install and reset. Finally, this divergence causes a having to separate and duplicated paths on the CI with heavy tests.

My suggestion is to use the common Live approach for the recovery image. That is embedding a filesystem container into a squashfs image. In former days (likely to be still the case for many distros) live iso included a squashfs image including a single filesystem image (ext2 or any other common FS) that gets loop mounted at boot time. This is handy to ensure root filesystem has a UUID, LABEL or any common filesystem attribute. IMHO if we assume that a recovery image is always a squashfs image which includes a filesystem image we can make both use cases to be really close (only the image content changes, nothing else) and in all cases we get the benefit of compressed recovery images.

Benefits:

• simplifies recovery image management code (single path)
• simplifies recovery system detection code
• simplifies grub configuration and user experience there
• reduces disk footprint (if using compressed squashfs, it can be uncompressed too)
• In the CI we are currently testing both paths, we could completely kill one, as both would be equivalent (it would free a lot of CI resources).

Counter parts:

• requires some refactor in elemental
• requries a very small refactor in cOS grub and immutablefs packages
• building a package including the squashfs has some additional constraints:
  • mkfs.ext2 -d <root> file.img should be used, it is unclear to me if the -d flag has some limitation. This is required to enable the build inside a container.

[mudler]

While I'm totally in favor of simplifying the whole logic in there, I think we are missing also the point that brings supporting both: downstream derivative can either pick to have an immutable recovery (squashfs) in their system (so, upgrade, reset, etc. act accordingly) or having a mutable one that can be also changed in runtime (for example, debugging, but also situations where recovery is not meant to be ro).

We have to take a call here and decide if we want to get rid of the latter use case.

Edit: Besides, we have also to consider that ISOs built with squashfs have also a bigger size currently, or do we take a snap of the booting one as recovery?

[davidcassany]

having a mutable one that can be also changed in runtime (for example, debugging, but also situations where recovery is not meant to be ro)

Correct, I did not think about this use case. It is currently possible, but not that obvious, debug kernel parameters are needed. Probably we can think about supporting everything and making the squashfs choice ortogonal to the recovery image contents. Here goes my straight forward three steps suggestion :smile::

1. Drop the current squashfs path (we could drop the alternate recovery image feature for the time being)
2. Add an squashfs opt in config option to embed recovery image (the ext2 formatted file) into a compressed squashfs container. (reduced recovery size at the cost of making it immutable)
3. Add support for alternate source for recovery images (form a docker image | channel | installation media | directory)

If we do the first step properly, the other two shouldn't be that complex. Note that 2 and 3 are not necessarily in that order.

[mudler]

having a mutable one that can be also changed in runtime (for example, debugging, but also situations where recovery is not meant to be ro)

Correct, I did not think about this use case. It is currently possible, but not that obvious, debug kernel parameters are needed. Probably we can think about supporting everything and making the squashfs choice ortogonal to the recovery image contents. Here goes my straight forward three steps suggestion smile:

1. Drop the current squashfs path (we could drop the alternate recovery image feature for the time being)

If I got it right this is just restricted in the elemental-cli installation context, but, at the same time changing slightly how we would load a squashfs if found with the new layout proposed above in the initrd/grub components. (correct me if I'm wrong)

2. Add an squashfs opt in config option to embed recovery image (the ext2 formatted file) into a compressed squashfs container. (reduced recovery size at the cost of making it immutable)

3. Add support for alternate source for recovery images (form a docker image | channel | installation media | directory)

If we do the first step properly, the other two shouldn't be that complex. Note that 2 and 3 are not necessarily in that order.

Yes right, at that point how we produce that squashfs can be exposed in various way, but the core on how it is produced could still be the same :+1:

[davidcassany]

at the same time changing slightly how we would load a squashfs if found with the new layout proposed above in the initrd/grub components.

This is what I would completely drop for now. So no ISOs including a recovery image and enforce the recovery image to be equivalent to the active one. Then we could think of how to enable a different recovery image (aka from recovery/cos), which could be always without squashfs and be a simple compressed file in an installation media (no need of squashfs). Finally we could think of embedding recovery into a squashfs if we want to make this partition way smaller by wrapping the whole ext2 filesystem, this way the change could be almost transparent to grub and initrd, as this this only implies a simple nested loop to find the expected filesystem.

[davidcassany]

In addition, I also realized that we could have the mutable recovery system by setting a proper persistent overlayfs, so we could eventually support mutable recovery with persistency and a recovery image bit a bit equivalent to the original active or original recovery (if it is from a different source). All in all, I think we can support all features and use cases with single and properly thought recovery setup and simplify a lot CI and upgrade and install procedures.

[mudler]

@davidcassany I guess this is partly already covered by the new elemental-cli features, right? now it can be specified during runtime if the recovery should be a squashfs or not.

Can we close this issue? or are there any points I'm missing?

[davidcassany]

Indeed a big chunk of the changes proposed here are done. Just the final step of my suggestion would be missing:

My suggestion is to use the common Live approach for the recovery image. That is embedding a filesystem container into a squashfs image. In former days (likely to be still the case for many distros) live iso included a squashfs image including a single filesystem image (ext2 or any other common FS) that gets loop mounted at boot time. This is handy to ensure root filesystem has a UUID, LABEL or any common filesystem attribute. IMHO if we assume that a recovery image is always a squashfs image which includes a filesystem image we can make both use cases to be really close (only the image content changes, nothing else) and in all cases we get the benefit of compressed recovery images.

This is essentially to converge all use cases at boot level, so we always use a filesystem label to identify the recovery image, regardless of being squashfs or not. This would simplify some recovery detection checks we have and also make our grub config smoother (aka, no weird error messages at boot around the use of squashfs or not).

Probably we can track that in another card with a devoted description and leave it around just as an idea to consider (or not) at some point.

[davidcassany]

No longer relevant and solved by slightly different approaches