linux
linux copied to clipboard
raspberrypi
Could save a bit of space by compressing kernel modules.
Describe the bug
The kernel has the option CONFIG_MODULE_COMPRESS_XZ=y (or another compression algorithm, if preferred) which compresses the modules when they're installed. Raspberry Pi OS doesn't seem to have trouble loading .ko.xz modules. Maybe it wouldn't be a bad idea to enable this by default.
Steps to reproduce the behaviour
N/A
Device (s)
Other
System
N/A
Logs
Stock:
60M /lib/modules/5.15.30+
62M /lib/modules/5.15.30-v7+
63M /lib/modules/5.15.30-v7l+
82M /lib/modules/5.15.30-v8+
265M total
With xz-compressed modules:
22M /lib/modules/5.15.30+
23M /lib/modules/5.15.30-v7+
23M /lib/modules/5.15.30-v7l+
24M /lib/modules/5.15.30-v8+
91M total
dmesg shows [ 0.000000] Memory: 3601320K/4050944K available (10240K kernel code, 1390K rwdata, 3324K rodata, 2048K init, 590K bss, 121944K reserved, 327680K cma-reserved, 3264512K highmem) in both cases.
No difference to kernel size.
I'm not seeing any negative impact to boot times.
Additional context
No response
I always wondered why RPi's modules were not compressed when seeing others doing it. I compiled the latest changes to 5.15.32/5.17.1 kernels today and enabled CONFIG_MODULE_COMPRESS_XZ=y and it did just fine on both kernels. Here is the boot times on the 5.15.32 with uncompressed and compressed modules. which surprised me a little:
Uncompressed:
[ray@pi4 ~]$ systemd-analyze
Startup finished in 2.396s (kernel) + 11.263s (userspace) = 13.660s
graphical.target reached after 11.262s in userspace
Compressed:
[ray@pi4 ~]$ systemd-analyze
Startup finished in 2.378s (kernel) + 11.119s (userspace) = 13.498s
graphical.target reached after 11.118s in userspace
I went ahead and released these kernels with the modules compressed to manjaro-arm's unstable repo which is the same as arch-arm's normal repo. If there is any issue it will be reported here after this post:
https://forum.manjaro.org/t/raspberry-pi-kernels-2-0/84885/413
Any reason not to have used CONFIG_MODULE_COMPRESS_ZSTD?
ZSTD decompression is IIRC an order of magnitude faster than XZ's decompression. See https://lists.archlinux.org/pipermail/arch-dev-public/2019-March/029542.html for some extensive comparisons (though those comparisons were not done on kernel module compression.)
(We have been using it in Chromebrew to compress our packages for the past several months, having moved to zstd from xz.)
@satmandu no reason, other than the experiments haven't been done for ZSTD, but have for XZ.
I'll look into it when I get time, although if you, or others can provide any information, such as systemd-analyze times for different Pi models and/or different boot mechanisms (sdcard, ssd, usb, nfs), that that may help make the best decision.
The systemd-analyze I reported above was using the pi4 booted on a usb ssd. All kernels I have released in the last 3 weeks to the manjaro-arm repo the modules has been compressed with xz and no one has reported an issue. I release 3 kernels each time: 5.15, and current RPi versions of upstream's "Stable" / "Mainline" kernels.
Yes, but the question is whether CONFIG_MODULE_COMPRESS_ZSTD has better boot times.
In past experiences zstd using it's default settings most likely decompress faster. Going with the topic of this issue to save space though we switched from .xz to .zst with our packages and found the package size increased. One could over ride the default zstd compression ratio and probably get close to to xz but then the question might be since it would be a tighter compression ratio how would it affect the boot time decompressing? I am guessing when building the kernel the default zstd settings would be used.
Looks like to me with out doing a lot of testing and the kernel I am guessing will use xz/zst defaults it could come down to do you want save some space and have the fasted boot time with zstd or save more space with most likely a slower boot time.
A good trade off might be go with zstd.
With our Chromebrew project we switched from xz to zstd for exactly that reason. There seems to be a ton of development happening with zstd, and it seems really well optimized for a variety of workloads, and for situations with multiple CPUs.
The tradeoff of slightly more space than xz for increases in decompression speed seemed to be a reasonable one.
I can't speak to the zstd compression settings used with the kernel, but I would assume that they are set to be widely compatible.
XZ. Pi4. force_turbo=0
Startup finished in 8.047s (kernel) + 11.963s (userspace) = 20.010s
graphical.target reached after 11.899s in userspace
$ find /usr/lib/modules/5.15.34-v7l+/kernel/ -name "*.xz" | xargs du -sch
20M total
ZSTD. Pi4. force_turbo=0
Startup finished in 8.052s (kernel) + 10.563s (userspace) = 18.615s
graphical.target reached after 10.247s in userspace
$ find /usr/lib/modules/5.15.34-v7l+/kernel/ -name "*.zst" | xargs du -sch
26M total
So there is a speed boost, but comes with a size increase.
XZ. Pi1. force_turbo=1
Startup finished in 9.435s (kernel) + 1min 3.335s (userspace) = 1min 12.770s
multi-user.target reached after 1min 350ms in userspace
$ find /usr/lib/modules/5.15.34+/kernel/ -name "*.xz" | xargs du -sch
19M total
ZSTD. Pi1. force_turbo=1
Startup finished in 9.459s (kernel) + 1min 7.632s (userspace) = 1min 17.091s
multi-user.target reached after 58.223s in userspace
$ find /usr/lib/modules/5.15.34+/kernel/ -name "*.zst" | xargs du -sch
24M total
Pi1 seems slower with ZSTD (surprisingly), as well as larger (as expected).
Seems like a wash for armv7l... (and the slower Pi1 results could be due to memory pressure?)
Is there any difference for armv8 where zstd might be more optimized?
XZ. Pi4. force_turbo=0. arm_64bit=1
Startup finished in 6.836s (kernel) + 10.423s (userspace) = 17.260s
multi-user.target reached after 10.381s in userspace
$ find /lib/modules/5.15.34-v8+/kernel/ -name "*.xz" | xargs du -sch
20M total
ZSTD. Pi4. force_turbo=0. arm_64bit=1
Startup finished in 6.932s (kernel) + 8.675s (userspace) = 15.607s
multi-user.target reached after 8.626s in userspace
$ find /lib/modules/5.15.34-v8+/kernel/ -name "*.zst" | xargs du -sch
26M total
Seems similar to 32-bit result.
With a two-second difference in boot time, it doesn't seem worth it to change to zstd at this time, right? Especially with 6Mb more space used for the compressed kernel modules? Thanks for your efforts in benchmarking this!
Yes, that's my opinion. I don't think the 10% (1.6s) speed boost compensates for the 30% size increase. If a user really cared then a self-compiled kernel could be used (and I suspect disabling unneeded options would gain much more).
The .xz compression stopping the kernel modules from being loaded. Not sure if the issue arised here because of the change or it inherit from other upstream layer. Please have a look here, https://github.com/agherzan/meta-raspberrypi/issues/1054
As I don't use meta-raspberrypi, can you explain if that is using RpiOS as the distribution? Is it using our pre-built kernel and modules from raspberrypi/firmware or raspberrypi/rpi-firmware? Is it building the kernel/modules from source with the raspberrypi/linux tree here?
I seem to be seeing the same thing with kernels I've built myself.
From the build, the module.* files in /lib/modules/
total 216
drwxrwxr-x 11 pi pi 4096 Apr 26 18:25 kernel
-rw-r--r-- 1 pi pi 45 Apr 26 18:25 modules.alias
-rw-r--r-- 1 pi pi 12 Apr 26 18:25 modules.alias.bin
-rw-rw-r-- 1 pi pi 14919 Apr 26 18:24 modules.builtin
-rw-r--r-- 1 pi pi 16644 Apr 26 18:25 modules.builtin.bin
-rw-rw-r-- 1 pi pi 84409 Apr 26 18:24 modules.builtin.modinfo
-rw-r--r-- 1 pi pi 0 Apr 26 18:25 modules.dep
-rw-r--r-- 1 pi pi 12 Apr 26 18:25 modules.dep.bin
-rw-r--r-- 1 pi pi 52 Apr 26 18:25 modules.devname
-rw-rw-r-- 1 pi pi 64188 Apr 26 18:24 modules.order
-rw-r--r-- 1 pi pi 55 Apr 26 18:25 modules.softdep
-rw-r--r-- 1 pi pi 49 Apr 26 18:25 modules.symbols
-rw-r--r-- 1 pi pi 12 Apr 26 18:25 modules.symbols.bin
Run sudo depmod -a, and I get sensibly sized files there, and the system loads modules fine.
pi@raspberrypi:/lib/modules/5.15.34-v7l+ $ ls -l
total 2476
drwxrwxr-x 11 pi pi 4096 Apr 26 18:25 kernel
-rw-r--r-- 1 root root 617829 Apr 26 18:28 modules.alias
-rw-r--r-- 1 root root 633004 Apr 26 18:28 modules.alias.bin
-rw-rw-r-- 1 pi pi 14919 Apr 26 18:24 modules.builtin
-rw-r--r-- 1 root root 0 Apr 26 18:28 modules.builtin.alias.bin
-rw-r--r-- 1 root root 16644 Apr 26 18:28 modules.builtin.bin
-rw-rw-r-- 1 pi pi 84409 Apr 26 18:24 modules.builtin.modinfo
-rw-r--r-- 1 root root 200967 Apr 26 18:28 modules.dep
-rw-r--r-- 1 root root 274856 Apr 26 18:28 modules.dep.bin
-rw-r--r-- 1 root root 324 Apr 26 18:28 modules.devname
-rw-rw-r-- 1 pi pi 64188 Apr 26 18:24 modules.order
-rw-r--r-- 1 root root 883 Apr 26 18:28 modules.softdep
-rw-r--r-- 1 root root 267016 Apr 26 18:28 modules.symbols
-rw-r--r-- 1 root root 324797 Apr 26 18:28 modules.symbols.bin
I don't know if the depmod list gets stashed somewhere else with compressed modules.
I will say that I haven't done a clean build after having CONFIG_MODULES_COMPRESS_XZ enabled, so it's possibly some dependency not being picked up with the config changed.
I also build with the object files being in a different directory to the source (using O=<directory> in the make command), so that may be confusing things.
I don't seem to be seeing this on my kernel builds. modules.dep is 200K.
dom@buildbot:~/projects/linux$ ls -l ../modules/lib/modules/5.15.35-v7l+/
total 2508
lrwxrwxrwx 1 dom dom 24 Apr 26 15:41 build -> /home/dom/projects/linux
drwxrwxr-x 11 dom dom 4096 Apr 26 15:54 kernel
-rw-r--r-- 1 dom dom 600581 Apr 26 15:54 modules.alias
-rw-r--r-- 1 dom dom 625103 Apr 26 15:54 modules.alias.bin
-rw-rw-r-- 1 dom dom 14919 Apr 26 15:53 modules.builtin
-rw-r--r-- 1 dom dom 28169 Apr 26 15:54 modules.builtin.alias.bin
-rw-r--r-- 1 dom dom 16644 Apr 26 15:54 modules.builtin.bin
-rw-rw-r-- 1 dom dom 84409 Apr 26 15:53 modules.builtin.modinfo
-rw-r--r-- 1 dom dom 217430 Apr 26 15:54 modules.dep
-rw-r--r-- 1 dom dom 292588 Apr 26 15:54 modules.dep.bin
-rw-r--r-- 1 dom dom 324 Apr 26 15:54 modules.devname
-rw-rw-r-- 1 dom dom 66013 Apr 26 15:53 modules.order
-rw-r--r-- 1 dom dom 883 Apr 26 15:54 modules.softdep
-rw-r--r-- 1 dom dom 264677 Apr 26 15:54 modules.symbols
-rw-r--r-- 1 dom dom 322327 Apr 26 15:54 modules.symbols.bin
lrwxrwxrwx 1 dom dom 24 Apr 26 15:53 source -> /home/dom/projects/linux
and this is a tree with compressed modules:
dom@buildbot:~/projects/linux$ ls -l ../modules/lib/modules/5.15.35-v7l+/kernel/arch/arm/crypto/
total 48
-rw-rw-r-- 1 dom dom 6664 Apr 26 15:53 aes-arm-bs.ko.xz
-rw-rw-r-- 1 dom dom 2112 Apr 26 15:53 aes-arm.ko.xz
-rw-rw-r-- 1 dom dom 2944 Apr 26 15:53 blake2s-arm.ko.xz
-rw-rw-r-- 1 dom dom 5236 Apr 26 15:53 chacha-neon.ko.xz
-rw-rw-r-- 1 dom dom 5584 Apr 26 15:53 curve25519-neon.ko.xz
-rw-rw-r-- 1 dom dom 4632 Apr 26 15:53 poly1305-arm.ko.xz
-rw-rw-r-- 1 dom dom 2644 Apr 26 15:53 sha1-arm.ko.xz
-rw-rw-r-- 1 dom dom 3308 Apr 26 15:53 sha1-arm-neon.ko.xz
As I don't use meta-raspberrypi, can you explain if that is using RpiOS as the distribution?
No, it is not using RpiOS. meta-raspberrypi is a yocto layer which is using bitbake as a build system and some bit of openembbed-core.
Is it using our pre-built kernel and modules from raspberrypi/firmware or raspberrypi/rpi-firmware?
Only precompiled raspberrypi/rpi-firmware is used as the sources are not available.
Is it building the kernel/modules from source with the raspberrypi/linux tree here?
Yes, Is it building the kernel/modules from source with the raspberrypi/linux tree here.
@bhargavthriler what is the size of your modules.dep file? What is the size after running sudo depmod -a?
Clean build on our Ubuntu 20.4 build server gives me a sensibly sized modules.dep file. arm-linux-gnueabihf-gcc is version 9.4.0
Clean build on my Ubuntu 16.04 laptop VM gives me a 0-byte sized modules.dep. arm-linux-gnueabihf-gcc is version 5.4.0
Suspect it's a tools version issue. What are you building with?
Ubuntu 22.04 appears to be building modules.dep correctly. Time for me to update to a new VM.
@bhargavthriler what is the size of your modules.dep file? What is the size after running
sudo depmod -a?
modules.dep size (initial boot) : 0
root@raspberrypi3:~# uname -a
Linux raspberrypi3 5.15.34-v7 #1 SMP Tue Apr 19 19:21:26 UTC 2022 armv7l GNU/Linux
root@raspberrypi3:/lib/modules/5.15.34-v7# ls -lh
drwxr-xr-x 8 root root 1.0K Mar 9 12:34 kernel
-rw-r--r-- 1 root root 45 Jan 1 1970 modules.alias
-rw-r--r-- 1 root root 12 Jan 1 1970 modules.alias.bin
-rw-r--r-- 1 root root 14.6K Mar 9 12:34 modules.builtin
-rw-r--r-- 1 root root 26.9K Jan 1 1970 modules.builtin.alias.bin
-rw-r--r-- 1 root root 16.0K Jan 1 1970 modules.builtin.bin
-rw-r--r-- 1 root root 84.1K Mar 9 12:34 modules.builtin.modinfo
-rw-r--r-- 1 root root 0 Jan 1 1970 modules.dep
-rw-r--r-- 1 root root 12 Jan 1 1970 modules.dep.bin
-rw-r--r-- 1 root root 0 Jan 1 1970 modules.devname
-rw-r--r-- 1 root root 63.3K Mar 9 12:34 modules.order
-rw-r--r-- 1 root root 55 Jan 1 1970 modules.softdep
-rw-r--r-- 1 root root 49 Jan 1 1970 modules.symbols
-rw-r--r-- 1 root root 12 Jan 1 1970 modules.symbols.bin
root@raspberrypi3:/lib/modules/5.15.34-v7# du -s *
1074 kernel
1 modules.alias
1 modules.alias.bin
15 modules.builtin
27 modules.builtin.alias.bin
17 modules.builtin.bin
85 modules.builtin.modinfo
0 modules.dep
1 modules.dep.bin
0 modules.devname
64 modules.order
1 modules.softdep
1 modules.symbols
1 modules.symbols.bin
modules.dep size after depmod -a : 0
root@raspberrypi3:/lib/modules/5.15.34-v7# depmod -a
root@raspberrypi3:/lib/modules/5.15.34-v7# du -s *
1074 kernel
1 modules.alias
1 modules.alias.bin
15 modules.builtin
27 modules.builtin.alias.bin
17 modules.builtin.bin
85 modules.builtin.modinfo
0 modules.dep
1 modules.dep.bin
0 modules.devname
64 modules.order
1 modules.softdep
1 modules.symbols
1 modules.symbols.bin
ah just saw comments in meta-raspberrypi , kmod doesn't have support for XZ will check some more comments and update.
@popcornmix can I get some reference/link on how modules.dep and depmod handle the kernel modules during boot ? As I can see after enabling XZ support in kmod the modules.deb is populated now (size : 2074) but the modules are not loaded. It only loads after executing depmod --all .
yocto is a cross build system and runs depmod wrapper on build host to generate the modules.dep file. So we need to ensure that native kmod can understand xz compressed modules and operate on them. If you built kmod for target and did depmod -a on target then you will get what all binary distros are getting here. The problem is in cross building the module dependency files.
