dracut

dracut copied to clipboard

Describe the bug We are getting a false error during the boot-time that dracut is not able to mount /sysroot. Although, the system continues to boot smoothly and I get the login shell. We are able to login and manipulate files.

Distribution used CentOS Stream 9

Dracut version We are using dracut-59

Init system Systemd

Full Picture

• Our goal is to boot a system over the network and mount a RAM-Disk based rootfilesystem (a squashfs image).
• We are booting vanilla linux kernel v5.10.152. There are no kernel modules as we have built all the necessary drivers inside the kernel (ex: squashfs).
• We added rd.info, and rd.shell. option in GRUB config file.
• We are building our initrd image inside a CentOS stream 9 based podman (version 4.2.0) image.

Initially we were using dracut-57, and we used following conf:

compress="gzip"
hostonly="no"
mdadmconf="no"
lvmconf="no"
nofscks="yes"
ro_mnt="no"
no_kernel="yes"
dracut_rescue_image="no"
#use_fstab="no"

dracutmodules+=" livenet "
dracutmodules+=" dracut-systemd "

And this puts following modules in it:

bash
systemd
systemd-initrd
systemd-sysusers
dbus-broker
dbus
network-manager
network
url-lib
dm
dmsquash-live
kernel-network-modules
livenet
overlayfs
rootfs-block
udev-rules
dracut-systemd
base
fs-lib
img-lib

Everything works till here.

---

But then we switched to dracut-59, with same configurations and same grub options; we get this false error during boot time:

[   40.845919] dracut-initqueue[475]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   40.970894] dracut-initqueue[475]:                                  Dload  Upload   Total   Spent    Left  Speed
100  152M  100  152M    0     0   111M      0  0:00:01  0:00:01 --:--:--  111M
[FAILED] Failed to mount /sysroot.
[   42.576025] dracut-initqueue[520]: Unknown device "/dev//tmp/curl_fetch_url0/image.squashfs": No such device
[   42.760858] dracut-initqueue[521]: Unknown device "/sys/": No such device
[   42.843726] dracut-initqueue[531]: modprobe: FATAL: Module squashfs not found in directory /lib/modules/5.10.152-rt75-fec-os-kernel-0.1.2

The system continues to boot; we are able to login, manipulate files, but still this error (Failed to mount /sysroot) is thrown at boot time. Also, as we can see, we get warning for module not found in /lib/modules/... directory, which should have been solved after this commit : https://github.com/dracutdevs/dracut/commit/922c9e28ed87815cf6ae0b5ee911ff0ef616d1b0.

We also had to add additional parameter for dracut-59 in grub.cfg : rd.live.overlay.overlayfs=1 ; even though we are providing kernel parameter as root=live:http://<location>.

---

So we added a breakpoint: rd.break=pre-pivot.

The output of journalctl from dracut shell:

sh-5.1# journalctl --no-pager
....
....
Feb 27 00:00:17 cfv-774-cdv28 kernel: overlayfs: failed to resolve '/run/overlayfs': -2
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: run-initramfs-squashfs.mount: Deactivated successfully.
Feb 27 00:00:17 cfv-774-cdv28 dracut-initqueue[517]: Unknown device "/dev//tmp/curl_fetch_url0/image.squashfs": No such device
Feb 27 00:00:17 cfv-774-cdv28 kernel: dracut-mount (555) used greatest stack depth: 13064 bytes left
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: Finished dracut initqueue hook.
Feb 27 00:00:17 cfv-774-cdv28 mount[552]: mount: /sysroot: special device LiveOS_rootfs does not exist.
Feb 27 00:00:17 cfv-774-cdv28 dracut-initqueue[518]: Unknown device "/sys/": No such device
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: Reached target Preparation for Remote File Systems.
Feb 27 00:00:17 cfv-774-cdv28 dracut-initqueue[528]: modprobe: FATAL: Module squashfs not found in directory /lib/modules/5.10.152-rt75-fec-os-kernel-0.1.2
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: Reached target Remote File Systems.
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: dracut pre-mount hook was skipped because no trigger condition checks were met.
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: Mounting /sysroot...
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: sysroot.mount: Mount process exited, code=exited, status=32/n/a
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: sysroot.mount: Failed with result 'exit-code'.
Feb 27 00:00:17 cfv-774-cdv28 dracut-pre-pivot[580]: Warning: Break before pre-pivot
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: Failed to mount /sysroot.
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: Reached target Initrd Root File System.
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: Starting Mountpoints Configured in the Real Root...
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: initrd-parse-etc.service: Deactivated successfully.
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: Finished Mountpoints Configured in the Real Root.

The output of mount from dracut shell:

sh-5.1# mount
none on / type rootfs (rw)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=999698,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=1605600k,nr_inodes=819200,mode=755)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
none on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
none on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/loop0 on /run/rootfsbase type squashfs (ro,relatime)
LiveOS_rootfs on /sysroot type overlay (rw,relatime,lowerdir=/run/rootfsbase,upperdir=/run/overlayfs,workdir=/run/ovlwork)

The output of systemctl from dracut shell.

sh-5.1# systemctl status sysroot.mount
WARNING: terminal is not fully functional
Press RETURN to continue
● sysroot.mount - /sysroot
     Loaded: loaded (/run/systemd/generator.early/sysroot.mount; generated)
     Active: active (mounted) since Mon 2023-02-27 00:00:17 UTC; 4min 7s ago
      Until: Mon 2023-02-27 00:00:17 UTC; 4min 7s ago
      Where: /sysroot
       What: LiveOS_rootfs
        CPU: 2ms

Feb 27 00:00:17 cfv-774-cdv28 mount[552]: mount: /sysroot: special device LiveO>
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: Mounting /sysroot...
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: sysroot.mount: Mount process exited, >
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: sysroot.mount: Failed with result 'ex>
Feb 27 00:00:17 cfv-774-cdv28 systemd[1]: Failed to mount /sysroot.