dockerd hangs initializing at "Loading containers: done."

[johndevedu]

Description

During initialization of dockerd, it hangs at "Loading containers: done." This is happening across multiple environments (AWS, esxi), but consistently in AWS EC2 with encrypted EBS. Same setup without encrypted EBS does not reproduce issue.

Reproduce

1. dockerd --debug
2. hangs and all docker commands hang i.e. docker ps docker info etc

Expected behavior

No response

docker version

Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.3
 Git commit:        24.0.5-0ubuntu1~22.04.1
 Built:             Mon Aug 21 19:50:14 2023
 OS/Arch:           linux/amd64
 Context:           default

Also tried 5:25.0.0-1~ubuntu.22.04~jammy but same result

docker info

Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Diagnostics ID

n/a

Additional Info

goroutine-stacks-2024-02-26T164610Z.log

[johndevedu]

Looks to me like this is the call stack that is responsible for the hang:

goroutine 1 [syscall, 15 minutes, locked to thread]:
syscall.Syscall6(0xc000d11ce0?, 0xc000e2b1e0?, 0xc000dbe97e?, 0xc000dbeaa0?, 0x55c538c76a4e?, 0xc000b39f70?, 0x1?)
        /usr/lib/go-1.20/src/syscall/syscall_linux.go:91 +0x36
os.(*Process).blockUntilWaitable(0xc0004ffe60)
        /usr/lib/go-1.20/src/os/wait_waitid.go:32 +0x87
os.(*Process).wait(0xc0004ffe60)
        /usr/lib/go-1.20/src/os/exec_unix.go:22 +0x28
os.(*Process).Wait(...)
        /usr/lib/go-1.20/src/os/exec.go:132
os/exec.(*Cmd).Wait(0xc000d11ce0)
        /usr/lib/go-1.20/src/os/exec/exec.go:890 +0x45
os/exec.(*Cmd).Run(0x2?)
        /usr/lib/go-1.20/src/os/exec/exec.go:590 +0x39
os/exec.(*Cmd).Output(0xc000d11ce0)
        /usr/lib/go-1.20/src/os/exec/exec.go:984 +0xc7
github.com/docker/docker/daemon.(*Daemon).fillPlatformInfo(0xc0009f26c0, 0xc000581000, 0xc000e45200)
        /build/docker.io-app-24.0.5/.gopath/src/github.com/docker/docker/daemon/info_unix.go:48 +0x38a
github.com/docker/docker/daemon.(*Daemon).SystemInfo(0xc0009f26c0)
        /build/docker.io-app-24.0.5/.gopath/src/github.com/docker/docker/daemon/info.go:68 +0x818
github.com/docker/docker/daemon.NewDaemon({0x55c53aed0258?, 0xc00015ad70}, 0xc000204f00, 0xc000a4f830, 0xc0000524e0)
        /build/docker.io-app-24.0.5/.gopath/src/github.com/docker/docker/daemon/daemon.go:1140 +0x3006
main.(*DaemonCli).start(0xc00061f240, 0xc000999490)
        /build/docker.io-app-24.0.5/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon.go:232 +0xae9
main.runDaemon(...)
        /build/docker.io-app-24.0.5/.gopath/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:14
main.newDaemonCommand.func1(0xc000004c00?, {0xc0002300f0?, 0x1?, 0x1?})
        /build/docker.io-app-24.0.5/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:39 +0x94
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc000004c00, {0xc00012c010, 0x1, 0x1})
        /build/docker.io-app-24.0.5/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:916 +0x862
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000004c00)
        /build/docker.io-app-24.0.5/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:1044 +0x3bd
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /build/docker.io-app-24.0.5/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:968
main.main()
        /build/docker.io-app-24.0.5/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:109 +0x18c

Following the source code, appears that this is the last line in moby code before hang: https://github.com/moby/moby/blob/v24.0.5/daemon/info_unix.go#L48

Any suggestions?