runc:[2:INIT] stuck - status D (disk sleep)

[rajivramanathan]

We are seeing processes stuck behind runc init. cat /proc/<pid>/status is showing the process to be in uninterruptible sleep and the load on the machines keeps rising behind this stuck process. It gets stuck forever (we've seen this for days) and the only way to recover is to reboot the machine.

Are there compatibility issues between runc v1.1.4 and RHAT 7.9 (kernel version: 3.10.0-1160.83.1.el7.x86_64)? As we have not seen this issue on our Ubuntu 20.04+ instances.

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.9 (Maipo)

$ uname -r 
3.10.0-1160.83.1.el7.x86_64

$ runc --version 
runc version 1.1.4
commit: v1.1.4-0-g5fd4c4d1
spec: 1.0.2-dev
go: go1.17.10
libseccomp: 2.5.4

$ cat /proc/31369/status
Name:	runc:[2:INIT]
Umask:	0022
State:	D (disk sleep)
Tgid:	31369
Ngid:	0
Pid:	31369
PPid:	31343
TracerPid:	0
Uid:	0	0	0	0
Gid:	0	0	0	0
FDSize:	64
Groups:	
.... <snip>