Rocky Linux 8.5: seems to be loading sshd too late to enter LUKS password

[ericfrey]

I am debugging this on a VM using Rocky Linux 8.5. It was installed using autopart --encrypt, which I understand means that all partitions are encrypted.

When I boot the VM and get to the LUKS password prompt the network is running (there is a console message indicating the network is up and I can ping the VM), but ssh does not seem to be running (port22: connection refused when I try to ssh to it). The network is started using kernel command line parameters (rd.neednet=1 ip=dhcp).

I added rd.break=pre-mount to help debug. The password prompt comes up before the exit into the shell. If I enter the password, booting continues and it exits at the pre-mount stage. At that stage root is not mounted (ls / shows what seems to be the initrd and the first line of the mount command says none at / type rootfs). There is also nothing in boot. I tried rd.break=pre-trigger and rd.break=initqueue. They are before the LUKS password request and networking is not started.

I looked at the order of the modules, and sshd seems to be started well before the modules that deal with encryption and mounting.

The password prompt gives the uuid of the root partition. I am using UEFI boot, and /boot and /boot/efi are on partitions that are not encrypted.

I saw the issue about lack of randomness, but this VM already gets /dev/urandom from the host. The host (also Rocky Linux 8.5) is physical and has plenty of stuff running on it so I don't think that is the issue. Also, typing random characters into the prompt does not cause sshd to start.

It seems that the sshd is not starting early enough in the boot process. Any thoughts on how to fix or debug this or what the problem is?

[gsauthof]

Are you able to execute journalctl, e.g. journalctl -u sshd.service, in the early boot environment?

Perhaps sshd logs something interesting there ...

Regarding randomness - I don't understand how it gets /dev/urandom from the host. sshd reads from /dev/random, IIRC. In any case, you could check the available entropy (in the early boot environment, on the VM guest) via /proc/sys/kernel/random/entropy_avail to rule that cause out.

[tuider]

As far as I can tell i was having the same issues on a freshly installed CentOS 7 VM with encrypted root.

I solved it with the following change since CentOS7 doesn't support prohibit-password :

--- /usr/lib/dracut/modules.d/46sshd/sshd_config	2023-01-16 08:48:06.000000000 -0500
+++ /usr/lib/dracut/modules.d/46sshd/sshd_config.new	2023-01-16 11:58:59.574364048 -0500
@@ -1,5 +1,5 @@
 SyslogFacility                  AUTHPRIV
-PermitRootLogin                 prohibit-password
+PermitRootLogin                 without-password
 AuthorizedKeysFile              .ssh/authorized_keys
 AuthenticationMethods           publickey
 UsePAM                          no

I'm not sure if Rocky Linux 8.5 also needs the deprecated version of the same parameter PermitRootLogin.

[gsauthof]

I solved it with the following change since CentOS7 doesn't support prohibit-password :

It doesn't?

At least according to sshd_config(5) on CentOS 7 it does support it:

PermitRootLogin
Specifies whether root can log in using ssh(1).  The argument
must be yes, prohibit-password, without-password,
forced-commands-only, or no.  The default is yes.

If this option is set to prohibit-password or without-password,
password and keyboard-interactive authentication are disabled for
root.

Similar wording in the CentOS 8 sshd_config(5) man page.

[wandhydrant]

I had exactly the same symptom on a current AlmaLinux 8.7. It worked earlier on, but as I'm rarely rebooting, I can't say when it broke. The machine can be pinged, but sshd only gets started once you pass the disk passphrase step. This is because of the sshd.service file:

# required on Ubuntu 20.04
After=systemd-tmpfiles-setup.service

and inside that service file, there is:

After=local-fs.target systemd-sysusers.service

So I commented out the first After= line above in /usr/lib/dracut/modules.d/46sshd/sshd.service, and it was working again.

[gsauthof]

Interesting.

However, AFAICS After= just specifies an ordering (in case those services are started at the same time) - whereas you need to use Wants= or Requires= to add an dependency. So perhaps such a dependency is also added on AlmaLinux 8.7.

FWIW, the situation is different on a Fedora 37 system of mine, there everything works fine:

initramfs-ssh:/root# systemctl list-dependencies sshd.service
sshd.service
● └─system.slice
initramfs-ssh:/root# echo $(for i in active  enabled; do systemctl is-$i systemd-tmpfiles-s
etup; done)
active static
grep 'Require\|Wants\|Before\|After' /usr/lib/systemd/system/systemd-tmpfiles-setup.service
After=-.mount systemd-sysusers.service systemd-journald.service
Before=sysinit.target
Before=shutdown.target initrd-switch-root.target
RemainAfterExit=yes
initramfs-ssh:/root# grep 'Require\|Wants\|Before\|After'  /usr/lib/systemd/system/sshd.service
Before=cryptsetup.target
After=systemd-tmpfiles-setup.service
initramfs-ssh:/root# systemctl show -p Wants local-fs.target 
Wants=
initramfs-ssh:/root# systemctl show -p Requires local-fs.target 
Requires=

NB: The ordering of systemd-tmpfiles-setup.service is different during late boot:

# grep 'Require\|Wants\|Before\|After' /usr/lib/systemd/system/systemd-tmpfiles-setup.service
After=local-fs.target systemd-sysusers.service systemd-journald.service
Before=sysinit.target
Before=shutdown.target initrd-switch-root.target
RemainAfterExit=yes

[gsauthof]

Ok, this is due to:

https://github.com/dracutdevs/dracut/blob/1843c16cafff005f798c379225e97dce459bc3dd/modules.d/06dbus-broker/module-setup.sh#L86-L92 https://github.com/dracutdevs/dracut/blob/1843c16cafff005f798c379225e97dce459bc3dd/modules.d/06dbus-daemon/module-setup.sh#L89-L95

So either AlmaLinux 8.7's dbus-daemon/broker dracut module looks different or there is no dracut dependency on the dbus-broker/daemon module.

On Fedora 37 the dbus module is pulled in by resolved/networkd/netwwor-manager - and dbus then pulls in dbus-broker or daemon.