destroypod hangs

[jcvenegas]

runtime version: 2.1.1 commit id: d7de93a13edc9eddf7d6417f20a6faa3f56136dc docker version: 1.12.6

Enviroment: Host (virtual machine with nested virtualization enabled): Clear Linux

host | virtual machine | clear-container

Virtual Machine ( host ) RAM : 15 GB CPUs : 1 Command to launch the VM(host): qemu-system-x86_64 -enable-kvm -m 16384 -cpu host -drive file=13970-clear.img,if=virtio,aio=threads -vnc :10 -net nic,model=virtio -net nic,model=virtio -net user, -bios clear.fd

Description: Docker rm -f container_id hags, looking at cc-oci-runtime logs shows that it is waiting for the command destroypod:

cc-oci-runtime log:

2017-03-15T23:37:03.802115Z:27681:cc-oci-runtime:debug:writing message data to proxy socket: {"id":"hyper","data":{"hyperName":"destroypod","data":{}}}

cc-proxy log:

I0315 23:24:42.265997   18659 proxy.go:77] [client #23] attach(containerId=b90d3c437d85990ad23711c898c2fe5798e9f9efa5456d5cf35e2d862728b434)
I0315 23:24:42.269947   18659 proxy.go:77] [client #23] hyper(cmd=destroypod, data={})
I0315 23:24:42.269960   18659 vm.go:114] [vm b90d3c43 hyperstart] pid 201 exit normally, status 0
I0315 23:24:42.279939   18659 vm.go:114] [vm b90d3c43 hyperstart] exec pid 202, pid 201
I0315 23:24:42.289939   18659 vm.go:114] [vm b90d3c43 hyperstart] can not find exec whose pid is 201
I0315 23:24:42.301940   18659 vm.go:114] [vm b90d3c43 hyperstart] hyper_loop epoll_wait 1
I0315 23:24:42.313939   18659 vm.go:114] [vm b90d3c43 hyperstart] hyper_handle_event get event 1, he 0x41f1b0, fd 3. ops 0x41f030
I0315 23:24:42.325944   18659 vm.go:114] [vm b90d3c43 hyperstart] hyper_handle_event event EPOLLIN, he 0x41f1b0, fd 3, 0x41f030
I0315 23:24:42.337938   18659 vm.go:114] [vm b90d3c43 hyperstart] hyper_channel_read
I0315 23:24:42.349938   18659 vm.go:114] [vm b90d3c43 hyperstart] hyper send type 14, len 4
I0315 23:24:42.361939   18659 vm.go:114] [vm b90d3c43 hyperstart] get length 10
I0315 23:24:42.373939   18659 vm.go:114] [vm b90d3c43 hyperstart] hyper send type 14, len 4
I0315 23:24:42.385939   18659 vm.go:114] [vm b90d3c43 hyperstart] 0 0 0 4 0 0 0 a 7b 7d 
I0315 23:24:42.397938   18659 vm.go:114] [vm b90d3c43 hyperstart]  hyper_channel_handle, type 4, len 10
I0315 23:24:42.409972   18659 vm.go:114] [vm b90d3c43 hyperstart] get DESTROYPOD message
I0315 23:24:42.419955   18659 vm.go:114] [vm b90d3c43 hyperstart] Sending SIGTERM
I0315 23:24:42.431954   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 202
I0315 23:24:42.443978   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 198
I0315 23:24:42.455949   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 194
I0315 23:24:42.469950   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 174
I0315 23:24:42.481941   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 139
I0315 23:24:42.493940   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 137
I0315 23:24:42.505939   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 118
I0315 23:24:42.517939   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 107
I0315 23:24:42.529939   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 104
I0315 23:24:42.541938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 103
I0315 23:24:42.558942   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 91
I0315 23:24:42.576945   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 90
I0315 23:24:42.588940   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 89
I0315 23:24:42.600939   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 88
I0315 23:24:42.612938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 87
I0315 23:24:42.624938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 86
I0315 23:24:42.636938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 85
I0315 23:24:42.648938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 84
I0315 23:24:42.660939   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 83
I0315 23:24:42.672938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 82
I0315 23:24:42.682940   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 81
I0315 23:24:42.694942   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 80
I0315 23:24:42.706939   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 79
I0315 23:24:42.718938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 78
I0315 23:24:42.730938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 77
I0315 23:24:42.742938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 76
I0315 23:24:42.754939   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 75
I0315 23:24:42.766938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 74
I0315 23:24:42.778938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 73
I0315 23:24:42.790938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 72
I0315 23:24:42.802938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 71
I0315 23:24:42.814941   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 70
I0315 23:24:42.826939   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 69
I0315 23:24:42.838938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 68
I0315 23:24:42.850938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 67
I0315 23:24:42.862938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 66
I0315 23:24:42.874938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 65
I0315 23:24:42.886938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 64
I0315 23:24:42.898938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 63
I0315 23:24:42.910938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 62
I0315 23:24:42.922938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 61
I0315 23:24:42.934938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 60
I0315 23:24:42.944938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 59
I0315 23:24:42.956938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 58
I0315 23:24:42.968947   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 57
I0315 23:24:42.980940   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 56
I0315 23:24:42.996943   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 34
I0315 23:24:43.006940   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 33
I0315 23:24:43.018941   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 30
I0315 23:24:43.030938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 29
I0315 23:24:43.042938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 28
I0315 23:24:43.054938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 27
I0315 23:24:43.065938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 26
I0315 23:24:43.077938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 25
I0315 23:24:43.089938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 24
I0315 23:24:43.101938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 23
I0315 23:24:43.113938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 22
I0315 23:24:43.125938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 21
I0315 23:24:43.137938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 20
I0315 23:24:43.149938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 19
I0315 23:24:43.161938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 18
I0315 23:24:43.173938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 17
I0315 23:24:43.183938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 16
I0315 23:24:43.193938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 15
I0315 23:24:43.205938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 14
I0315 23:24:43.217938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 13
I0315 23:24:43.229938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 12
I0315 23:24:43.241938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 11
I0315 23:24:43.253938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 10
I0315 23:24:43.265938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 9
I0315 23:24:43.277938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 8
I0315 23:24:43.289937   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 7
I0315 23:24:43.301937   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 6
I0315 23:24:43.313938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 5
I0315 23:24:43.323938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 4
I0315 23:24:43.335937   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 3
I0315 23:24:43.347938   18659 vm.go:114] [vm b90d3c43 hyperstart] kill process 2
I0315 23:24:43.359938   18659 vm.go:114] [vm b90d3c43 hyperstart] fopen /proc/202/status
I0315 23:24:43.383938   18659 vm.go:114] [vm b90d3c43 hyperstart] find sigign 	0000000000001000
I0315 23:24:43.395939   18659 vm.go:114] [vm b90d3c43 hyperstart] mask is 1000
I0315 23:24:43.407961   18659 vm.go:114] [vm b90d3c43 hyperstart] pid 202 exit by signal, status 15
I0315 23:24:43.419960   18659 vm.go:114] [vm b90d3c43 hyperstart] exec pid 202, pid 202
I0315 23:24:43.431941   18659 vm.go:114] [vm b90d3c43 hyperstart] hyper_handle_exec_exit exec exit pid 202, seq 1, container b90d3c437d85990ad23711c898c2fe5798e9f9efa5456d5cf35e2d862728b434
I0315 23:24:43.443944   18659 vm.go:114] [vm b90d3c43 hyperstart] container init process 202
I0315 23:24:43.450113   18659 vm.go:114] [vm b90d3c43 hyperstart] still have 3 user of exec

[jcvenegas]

if we run the host (that is a VM) with more than one core, the issues goes away, so looks like something racy in hyperstart.

[gtkramer]

Can you post the workaround you used for starting a VM with QEMU?

[jcvenegas]

we tested with smp option from qemu to use 6 cpus: smp [cpus=]n - > -spm 6 And 8Gb for the VM: -m 8G

[jodh-intel]

@jcvenegas - any update? Has this issue be raised with hyperstart?