mkimg: Optimize boot times & initramfs build times

[LekKit]

By default, mkinitcpio decompresses kernel modules into a temporary directory when building initramfs, then compresses the resulting initramfs fully as a single blob

This default behavior might have been faster on some beefy x86_64 CPU, and also in the past when firmware wasn't compressed, we didn't have ZSTD etc, but in modern RISC-V reality it slows things down big time, because now the kernel has to decompress the whole huge initramfs at once upon initial boot instead of separately decompressing individual modules/firmware blobs. It doesn't help that firmware is seemingly compressed twice...

This patch to the mkinitcpio config shaves around 30 seconds of boot time on QEMU on my machine, and around 20 seconds of boot time on RVVM. Results from real boards are welcome (But I've definitely seen they also suffer from this slow initramfs decompression issue). Perhaps it is possible to patch the default Arch RISC-V mkinitcpio package config instead, but I guess this is a less invasive proposal for those who might be interested in optimizing boot time right now

I'd also like to see some way to omit linux-firmware from being installed in the resulting image, but I am not sure how to implement it properly or what arg syntax would be acceptable. Thx for any advice or help.

[CoelacanthusHex]

This default behavior might have been faster on some beefy x86_64 CPU, and also in the past when firmware wasn't compressed, we didn't have ZSTD etc, but in modern RISC-V reality it slows things down big time, because now the kernel has to decompress the whole huge initramfs at once upon initial boot instead of separately decompressing individual modules/firmware blobs.

Have you compared the initrd size? We have met too big initrd problem on D1 device before...

Perhaps it is possible to patch the default Arch RISC-V mkinitcpio package config instead, but I guess this is a less invasive proposal for those who might be interested in optimizing boot time right now

But it's more dirty...

[LekKit]

Have you compared the initrd size? We have met too big initrd problem on D1 device before...

1. There is no real difference on the disk. It is compressed either way, just that it's compressed in smaller individually usable pieces (modules/firmware blobs) instead of compressing the huge initramfs blob. My final observations were 17M for usual initramfs and 65M for a fallback one (However I didn't install linux-firmware as I don't need it).
2. It actually fixes your D1 issue! When we compress the whole initramfs it is actually very heavy on memory - amount of RAM used is compressed initramfs + fully decompressed initramfs at the early stages of boot which is easily more than 512MiB. When we use my approach RAM usage is never much higher than the on-disk initramfs size (See above numbers).

I can easily boot fallback initrams on 256M RAM given to the VM.

Any lower and U-Boot crashes however - it's a QEMU U-Boot config so it's really not configured to work below that (btw if anyone can help me with this, I'd be very thankful). If not for the U-Boot issue, I imagine we could boot default initramfs on somewhere around 64-128M of RAM.

But it's more dirty...

Yes. If you are impressed with the results as much as I am, hopefully we can ask for a package config change instead.

[CoelacanthusHex]

Have you compared the initrd size? We have met too big initrd problem on D1 device before...

1. There is no real difference on the disk. It is compressed either way, just that it's compressed in smaller individually usable pieces (modules/firmware blobs) instead of compressing the huge initramfs blob. My final observations were 17M for usual initramfs and 65M for a fallback one (However I didn't install linux-firmware as I don't need it).

If that will have no visible disk size change, I think this change should be sent to mkinitcpio upstream as default config. Because it's a Pareto Improvement.

[LekKit]

If that will have no visible disk size change, I think this change should be sent to mkinitcpio upstream as default config. Because it's a Pareto Improvement.

I am not entirely sure they will accept this right away and am shy. However, I can also see a few second reduction in boot times on my laptop with an Intel CPU and NVMe.

I will try to send such a patch upstream. If they won't accept it, I hope we can implement this in Arch RISC-V package.

[LekKit]

Sent MR upstream: https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/merge_requests/344