Suddenly, very long booting time

[CodeAnthem]

Hello, I am trying to solve my issue since hours and can't figure it out, that's why I seek out for help.

The issue When I reboot my server, it takes about 10-16min to boot, actually you can see the usual initramfs screen with network config and the prompt to enter luks password and usually I could unlock the server remotely. But it takes a long time, till the server gets pingable. Once it is, everything is normal. The server wasn't rebooted for a while 160days+ and I did some updates frequently

Infos

• OS: Debian 10 5.10.0-0.bpo.12-amd64
• Dropbear version: dropbear/oldstable,now 2018.76-5 all
• Raid 1 LUKS encrypted

Logs

• systemd-analyze Startup finished in 8.621s (firmware) + 374ms (loader) + 16min 50.537s (kernel) + 11.406s (userspace) = 17min 10.940s

• dmesg -T

[Di Jun 28 12:40:13 2022] xor: automatically using best checksumming function   avx
[Di Jun 28 12:40:13 2022] async_tx: api initialized (async)
[Di Jun 28 12:40:13 2022] device-mapper: uevent: version 1.0.3
[Di Jun 28 12:40:13 2022] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: [email protected]
[Di Jun 28 12:40:13 2022] process '/usr/bin/ipconfig' started with executable stack
[Di Jun 28 12:40:16 2022] e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[Di Jun 28 12:40:16 2022] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s31f6: link becomes ready
[Di Jun 28 12:56:58 2022] NET: Registered protocol family 38
[Di Jun 28 12:56:59 2022] Btrfs loaded, crc32c=crc32c-intel
[Di Jun 28 12:56:59 2022] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
[Di Jun 28 12:56:59 2022] e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Down
[Di Jun 28 12:56:59 2022] Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist.
[Di Jun 28 12:56:59 2022] systemd[1]: Inserted module 'autofs4'
[Di Jun 28 12:56:59 2022] systemd[1]: systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[Di Jun 28 12:56:59 2022] systemd[1]: Detected architecture x86-64.

What I already tried

• Jumping back to older kernel 10 5.10.0-0.bpo.11-amd64
• Disabling encrypted SWAP
• Removing hostname from initramfs config
• Switching to second network adapter (it was another one before)
• Reading logs and doing a lot of research trying to understand where the issue is

Configs /etc/default/dropbear

# disabled because OpenSSH is installed
# change to NO_START=0 to enable Dropbear
NO_START=1
# the TCP port that Dropbear listens on
DROPBEAR_PORT=22

# any additional arguments for Dropbear
DROPBEAR_EXTRA_ARGS=

# specify an optional banner file containing a message to be
# sent to clients before they connect, such as "/etc/issue.net"
DROPBEAR_BANNER=""

# RSA hostkey file (default: /etc/dropbear/dropbear_rsa_host_key)
#DROPBEAR_RSAKEY="/etc/dropbear/dropbear_rsa_host_key"

# DSS hostkey file (default: /etc/dropbear/dropbear_dss_host_key)
#DROPBEAR_DSSKEY="/etc/dropbear/dropbear_dss_host_key"

# ECDSA hostkey file (default: /etc/dropbear/dropbear_ecdsa_host_key)
#DROPBEAR_ECDSAKEY="/etc/dropbear/dropbear_ecdsa_host_key"

# Receive window size - this is a tradeoff between memory and
# network performance
DROPBEAR_RECEIVE_WINDOW=65536

/etc/dropbear-initramfs/config

#
# Configuration options for the dropbear-initramfs boot scripts.
# You must run update-initramfs(8) to effect changes to this file (like
# for other files under the '/etc/dropbear-initramfs' directory).

#
# Command line options to pass to dropbear(8)
#
DROPBEAR_OPTIONS="-I 180 -j -k -p 59876 -s -T 3 -R"

#
# On local (non-NFS) mounts, interfaces matching this pattern are
# brought down before exiting the ramdisk to avoid dirty network
# configuration in the normal kernel.
# The special value 'none' keeps all interfaces up and preserves routing
# tables and addresses.
#
#IFDOWN=*

/etc/initramfs-tools/initramfs.conf

# initramfs.conf
# Configuration file for mkinitramfs(8). See initramfs.conf(5).
#
# Note that configuration options from this file can be overridden
# by config files in the /etc/initramfs-tools/conf.d directory.

#
# MODULES: [ most | netboot | dep | list ]
#
# most - Add most filesystem and all harddrive drivers.
#
# dep - Try and guess which modules to load.
#
# netboot - Add the base modules, network modules, but skip block devices.
#
# list - Only include modules from the 'additional modules' list
#

MODULES=most

#
# BUSYBOX: [ y | n | auto ]
#
# Use busybox shell and utilities.  If set to n, klibc utilities will be used.
# If set to auto (or unset), busybox will be used if installed and klibc will
# be used otherwise.
#

BUSYBOX=auto

#
# KEYMAP: [ y | n ]
#
# Load a keymap during the initramfs stage.
#

KEYMAP=n

#
# COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz ]
#

COMPRESS=gzip

#
# NFS Section of the config.
#

#
# DEVICE: ...
#
# Specify a specific network interface, like eth0
# Overridden by optional ip= or BOOTIF= bootarg
#

DEVICE=enp0s31f6
IP=10.2.0.69::10.2.0.1:255.255.255.0::enp0s31f6:off

#
# NFSROOT: [ auto | HOST:MOUNT ]
#

NFSROOT=auto

#
# RUNSIZE: ...
#
# The size of the /run tmpfs mount point, like 256M or 10%
# Overridden by optional initramfs.runsize= bootarg
#

RUNSIZE=10%

grub

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=0
GRUB_DISABLE_OS_PROBER=true
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash video=SVIDEO-1:d"
GRUB_CMDLINE_LINUX=""

GRUB_GFXMODE=1280x800x24
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
GRUB_DISABLE_SUBMENU=y

Thanks a lot for reading, any help is apprechiated.

[CodeAnthem]

Can someone help?

[mkj]

From the timestamps it looks like something is delayed in the kernel booting, before any userspace programs run?

[Di Jun 28 12:40:16 2022] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s31f6: link becomes ready
[Di Jun 28 12:56:58 2022] NET: Registered protocol family 38

family 38 is AF_ALG so crypto accelerator - maybe there's something wrong with that driver for the hardware. Can you disable that?

[CodeAnthem]

Hello, thanks for answering me.

I''ve figured, that maybe its something with the IP lease. I had the server offline for about 3h. When I started it, it immediately was reachable. Like dropbear was reachable within 20sec.

[mkj]

Closing this, let me know if it still seems to be Dropbear-specific