Profile and reduce host OS image size

[willnewton]

Profile the on-disk size of resinOS and find areas where it can be reduced.

[shaunmulligan]

maybe look at some of the useless kernel modules we add by default, like PS/2 mouse and joystick drivers :P

[willnewton]

This is the list of installed packages. Total size is 218334kB, of which:

• docker is 27342kB
• linux-firmware-* is 50756kB
• kernel-module* is 44091kB
• The rest is 96137kB

Of the rest a large part of that is pulled in by systemd/networkmanager I think.

A good place to start would be slimming down firmwares for devices that do not need them, e.g. the Intel Wifi firmwares (40MB of them) don't make a whole lot of sense on non-Intel platforms.

Other things that stand out from the list that could be removed:

• kernel-module-ocfs2 (1230kB)
• lvm2 (4290kB + dependencies) do we need LVM? at least on rpi type boards I would think not
• dhcp-client (2078kB) we probably have a DHCP client in systemd and busybox, do we need this one?
• systemd-analyze (1223kB) should be restricted to dev images?
• libgnutls30 (1198kB + dependencies) for some reason yocto likes to build things (e.g. wpa_supplicant, curl) with gnutls, would be good to use openssl instead if possible
• bash (806kB) it is reasonably large but we have quite a lot of bash scripts...
• kernel-module-drbd (487kB)
• kernel-module-gfs2 (412kB)
• Other filesystems, e.g. cifs, nfs (may be useful for NAS applications), reiserfs, ntfs, jfs, hfsplus, udf all in the 100s of kB range

installed-package-sizes.txt

[willnewton]

It looks like the lvm2 dependency is dropped by the hostapps PR:

https://github.com/resin-os/meta-resin/pull/805/commits/f5c783df899172d9e904818835a6da575d6631ab

[willnewton]

Opened a PR to reduce kernel size here:

https://github.com/resin-os/meta-resin/pull/813

[imrehg]

@willnewton would the file systems make sense to keep since people can use external storage, etc? (and they did use, e.g. external USB stick and such) Or if those are removed from here, would people be able to add them back easily in the container?

[willnewton]

Opened a PR for systemd-analyze here:

https://github.com/resin-os/meta-resin/pull/814

[willnewton]

@imrehg the fileystems are mostly for servers or osbolete:

• ocfs2 - cluster filesystem
• gfs2 - cluster filesystem
• jfs - obsolete AIX filesystem
• hfs - Obsolete Apple filesystem
• hfs+ - More modern Apple filesystem but still rare for removable disks
• ntfs - Partial read-only support for Windows NTFS
• udf - DVD filesystem support
• reiserfs - No longer developed AFAIK

I left in NFS and CIFS as these seem legitimately useful, we also keep XFS.

[alexgg]

The comments above have size optimized the following:

• Kernel configuration
• Running services
• Firmware files

Although the effort above can be repeated, it is unlikely that we would find any big space saver left. The analysis for v2.48 is summarized in:

284M    /build/balena-intel/build/tmp/work/genericx86_64-poky-linux/resin-image/1.0-r0/rootfs/

36M     /lib/modules
61M     /lib/firmware
45M     /usr/lib
34.5M   /usr/bin/balena-engine
12.2M   /boot/bzImage
7.2M    /boot/init
6.4M    /resin-boot
5M      /usr/bin/os-config
3.1M    /usr/lib/rust/libstd.so
2.8M    /usr/sbin/NetworkManager
1.5M    /usr/sbin/ModemManager
1.4M    /lib/systemd/systemd
1M      /bin/bash.bash

218M, 67M unaccounted (< 1M objects)

On v2.51 with the update of balena-engine the filesystem has gone above it's threshold again. Looking at the above list the following actions can be considered:

1. Enable kernel compression

• Saves 23M
• Works on the supported Alpine and Fedora images
• Only works from Debian Buster and Ubuntu Cosmic

2. Enable firmware compression

• Requires a kernel >= v5.3

3. Enable balena-engine compression

• Saves 33M
• Increases boot time by 5s in Intel NUC

4. Enable os-config compression

• Saves 4M
• Increases boot time in less than 1s

5. Removing unneeded files from /boot directory

• Saves around 600K for grub based devices, 400K for u-boot based devices

Also, the inactive partition is 300M, but a 5% is reserved by default (~15M) so only a hostapp of 285M can be updated. As the root filesystem is read-only and nothing read or writes to it, there is no reason not to remove this reserve and use the whole available space, which is 300M minus the 8M journal size.

[alexgg]

Updating on this, according to https://wiki.debian.org/LTS Strech has gone EOL on the 30th June 2022. And according to https://ubuntu.com/about/release-cycle 18.04 LTS (Bionic Beaver) is now under ESM (Extended Security Maintenance). It's a good moment to explore adding module compression as default and checking that Stretch and Bionic can still use non-compressed kernel modules.

[jellyfish-bot]

[alexgg] This has attached https://jel.ly.fish/6e6207ca-a304-4b94-88fe-0cb6ddd40b2d

[alexgg]

Bullseye supports module compression:

root@87d09b8ecf24:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@87d09b8ecf24:/# modprobe -V
kmod version 28
-ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL