hyperhq
Bug report: leaking container while doing pressure test
- Test case:
- Test runv-containerd with docker daemon
- start 1000 container, then "docker rm" all
- Expected result: All containers are removed and no container is left
- Actual result: Some containers can't be removed and left on system forever, unless you restart docker daemon.
$ docker rm -f ea3bcc75e63a
Error response from daemon: Could not kill running container ea3bcc75e63adb0ad33fe59fc733af6cac9f6f484dabd33b3e798af3f458250e, cannot remove - Cannot kill container ea3bcc75e63adb0ad33fe59fc733af6cac9f6f484dabd33b3e798af3f458250e: rpc error: code = 2 desc = "The container ea3bcc75e63adb0ad33fe59fc733af6cac9f6f484dabd33b3e798af3f458250e or the process init is not found"
- Frequency: sometimes, not always. If you can't reproduce, then test again.
- Root cause: https://github.com/hyperhq/runv/blob/master/supervisor/hyperpod.go#L415
c.run(p) start the VM and container in a goroutine, but never return the error, so if the VM start failed, runv-containerd still send success response to docker, docker will regard this as a running container but it's not. That's why docker can't kill it and can't remove it any more, because it's trying to kill nothing!
When I try to dig deeper, I found that sometimes https://github.com/hyperhq/runv/blob/master/supervisor/container.go#L35 will hang, I can't find the real cause of the hanging, PLEASE find it, it's really vital!
one extra thing:
https://github.com/hyperhq/runv/blob/master/supervisor/container.go#L71 should return an error but not nil, I'll send a patch for this.
Note: I was testing based on our internal version which diverge a little bit with the latest upstream version, but I believe the problem is still there
When I try to dig deeper, I found that sometimes https://github.com/hyperhq/runv/blob/master/supervisor/container.go#L35 will hang
I found this too when integrate cri-o, it seems runv will hang when starting failed before container could start.
