Make qvm-template use less space for template installation

[marmarek]

How to file a helpful issue

The problem you're addressing (if any)

Currently qvm-template install can run out of disk space in dom0 (in /var/tmp specifically), examples:

• https://github.com/QubesOS/qubes-issues/issues/8675
• https://forum.qubes-os.org/t/what-is-the-latest-solution-for-dom0-qvm-template-no-space-on-device-problem-as-of-2023-under-r4-1/19445/3
• https://forum.qubes-os.org/t/cant-install-centos8-template/10281

This is because qvm-template currently keeps two copies of the template during installation:

• downloaded template rpm (this one is compressed)
• extracted template rpm in /var/tmp

and only then it writes actual root.img into target storage pool (LVM, btrfs etc), where more space is available.

The solution you'd like

Modify qvm-template (and qvm-template-postprocess) to not store intermediate root.img in /var/tmp, but instead write directly to the target volume. This might require a bit of refactor how qvm-template works. Currently it is:

1. qvm-tempate downloads the package, extract rpm content into temporary directory and calls qvm-template-postprocess on it. Part of the rpm package is a tar archive of the root.img, that is split into 1GB files:

$ qvm-template download --enablerepo=qubes-templates-community-testing centos-stream-8-xfce
Downloading 'qubes-template-centos-stream-8-xfce-0:4.2.0-202311250849'...
qubes-template-centos-stream-8-xfce-0:4.2.0-202311250849: 
...
$ rpm -qpl qubes-template-centos-stream-8-xfce-0\:4.2.0-202311250849.rpm 
/var/lib/qubes/vm-templates/centos-stream-8-xfce
/var/lib/qubes/vm-templates/centos-stream-8-xfce/apps
/var/lib/qubes/vm-templates/centos-stream-8-xfce/apps.tempicons
/var/lib/qubes/vm-templates/centos-stream-8-xfce/apps.templates
/var/lib/qubes/vm-templates/centos-stream-8-xfce/clean-volatile.img.tar
/var/lib/qubes/vm-templates/centos-stream-8-xfce/netvm-whitelisted-appmenus.list
/var/lib/qubes/vm-templates/centos-stream-8-xfce/private.img
/var/lib/qubes/vm-templates/centos-stream-8-xfce/root.img
/var/lib/qubes/vm-templates/centos-stream-8-xfce/root.img.part.00
/var/lib/qubes/vm-templates/centos-stream-8-xfce/root.img.part.01
/var/lib/qubes/vm-templates/centos-stream-8-xfce/root.img.part.02
/var/lib/qubes/vm-templates/centos-stream-8-xfce/root.img.part.03
/var/lib/qubes/vm-templates/centos-stream-8-xfce/template.conf
/var/lib/qubes/vm-templates/centos-stream-8-xfce/vm-whitelisted-appmenus.list
/var/lib/qubes/vm-templates/centos-stream-8-xfce/volatile.img
/var/lib/qubes/vm-templates/centos-stream-8-xfce/whitelisted-appmenus.list

2. qvm-template-postprocess takes this directory, extracts combined root.img.* files directly into the target storage, and also processes other files (template.conf, whitelisted-appmenus.list etc). Those extra files probably need to stay as separate extracted files.

Due to the extra files, I think the rpm2archive needs to be called twice:

1. once to extract everything except root.img.* for later processing by qvm-template-postprocess
2. and another time to get root.img.* and pipe it directly to further import process (it will be more or less rpm2archive -> tar -> tar -> import_data_with_size(), yes, really double tar)

Structure-wise, one option is for the qvm-template to symlink downloaded rpm somewhere into that temp directory (as template.rpm? or root.rpm?) and then the second rpm2archive can be put into yet another case in import_root_img() when that template.rpm is found, instead of processing already extracted root.img. There are already a few cases there, so it's easy to add another one. But I'm open for other designs too.

The value to a user, and who that user might be

Easier to install templates, especially big ones. The full Kali Linux template is not built currently, because it's so big that is impossible to install with default 20GB root partition in dom0...

[DemiMarie]

With the current signing scheme, the signature cannot be verified until the entire RPM has been downloaded. This means that streaming processing cannot be done securely.

An alternative is to use a data format that allows incremental verification. OpenBSD’s signify supports this.

[marmarek]

With the current signing scheme, the signature cannot be verified until the entire RPM has been downloaded. This means that streaming processing cannot be done securely.

But that's not what this ticket is about. The proposal here is to download RPM in full first, verify it, and then skip its second, extracted copy and use pipe at that stage.

[DemiMarie]

With the current signing scheme, the signature cannot be verified until the entire RPM has been downloaded. This means that streaming processing cannot be done securely.

But that's not what this ticket is about. The proposal here is to download RPM in full first, verify it, and then skip its second, extracted copy and use pipe at that stage.

Ah, I see. I misunderstood.

[slayoo]

For the record, this is being worked on here: https://github.com/QubesOS/qubes-core-admin-client/commit/b6eb7737e3062e1973f1702aa447a60a5f467b64

[slayoo]

FWIW (probably obvious, but since we discussed it over e-mail, just leaving a trace):

• the new logic assumes part files are in order in the tar
• the tar is created by rpm utils from a .spec file in which a part.* glob pattern is used
• spec file docs mention that Shell Globbing: The usual rules for shell globbing apply (see `glob(7)`): https://github.com/rpm-software-management/rpm/blob/ebece0fb63f7e68801b5c696a45a9957db5dcfa1/docs/manual/spec.md?plain=1#L766
• glob manpage (https://manned.org/glob.3) mentions that By default, the returned pathnames are sorted...
• ... which is actually required by POSIX (pattern shall be replaced with those filenames and pathnames, sorted according to the collating sequence in effect in the current locale): https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_13_03

[marmarek]

Is this ready to open a PR? If so, can you open it?

[slayoo]

finishing up tests, aiming at a PR today here's the lack-of-symlink case handled: https://github.com/QubesOS/qubes-core-admin-client/commit/e33eaf0ae0c0b541b4aefbaa1d51eba0c438a947

[slayoo]

Is this ready to open a PR? If so, can you open it?

Here it is: https://github.com/QubesOS/qubes-core-admin-client/pull/278

[slayoo]

for the record, the PR was just merged

[andrewdavidwong]

Closing as completed. If anyone believes this issue is not yet completed, or if anyone is still affected by this issue, please leave a comment, and we'll be happy to reopen it. Thank you.