cc-runtime fails to delete pod using kubectl after nginx workload finishes

[chavafg]

I ran bats nginx.bats using latest cc-runtime with kubernetes 1.7.5 (also happens with 1.6.7) and when the nginx deployment is deleted, the cc-runtime fails to delete the created pods.

kubernetes version:

fuentess@singlevm:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler
:"gc", Platform:"linux/amd64"}

cc-runtime version:

cc-runtime  : 3.0.0-beta.2
   commit   : f8f8c6ca784b19b52ba2c5fb0e3a145da264d463
   OCI specs: 1.0.0-rc5

kubelet.service log

Sep 18 22:58:49 singlevm kubelet[28005]: E0918 22:58:49.316538   28005 remote_runtime.go:126] RemovePodSandbox "4a5210c5f5c2fb30a372d3206b57843f880fc748d71cd8a685585f64ff411628" from runtime service failed: rpc
error: code = 2 desc = failed to delete container k8s_POD_nginx-service-103607745-svjl1_default_d4ab290f-9cc4-11e7-9249-525400123456_0 in pod sandbox 4a5210c5f5c2fb30a372d3206b57843f880fc748d71cd8a685585f64ff411
628: `/usr/local/bin/cc-runtime delete --force 4a5210c5f5c2fb30a372d3206b57843f880fc748d71cd8a685585f64ff411628` failed: No multicast available for CTL channel
Sep 18 22:58:49 singlevm kubelet[28005]:   (exit status 1)
Sep 18 22:58:49 singlevm kubelet[28005]: E0918 22:58:49.316724   28005 kuberuntime_gc.go:142] Failed to remove sandbox "4a5210c5f5c2fb30a372d3206b57843f880fc748d71cd8a685585f64ff411628": rpc error: code = 2 desc
 = failed to delete container k8s_POD_nginx-service-103607745-svjl1_default_d4ab290f-9cc4-11e7-9249-525400123456_0 in pod sandbox 4a5210c5f5c2fb30a372d3206b57843f880fc748d71cd8a685585f64ff411628: `/usr/local/bin
/cc-runtime delete --force 4a5210c5f5c2fb30a372d3206b57843f880fc748d71cd8a685585f64ff411628` failed: No multicast available for CTL channel
Sep 18 22:58:49 singlevm kubelet[28005]:   (exit status 1)
Sep 18 22:58:49 singlevm kubelet[28005]: E0918 22:58:49.465839   28005 remote_runtime.go:126] RemovePodSandbox "19a753f0dfc194e313ce8f77070660c208a88d1e30784c8ad913b0e970c94153" from runtime service failed: rpc
error: code = 2 desc = failed to delete container k8s_POD_nginx-service-103607745-53qlv_default_d4ab2fd8-9cc4-11e7-9249-525400123456_0 in pod sandbox 19a753f0dfc194e313ce8f77070660c208a88d1e30784c8ad913b0e970c94
153: `/usr/local/bin/cc-runtime delete --force 19a753f0dfc194e313ce8f77070660c208a88d1e30784c8ad913b0e970c94153` failed: No multicast available for CTL channel
Sep 18 22:58:49 singlevm kubelet[28005]:   (exit status 1)
Sep 18 22:58:49 singlevm kubelet[28005]: E0918 22:58:49.465954   28005 kuberuntime_gc.go:142] Failed to remove sandbox "19a753f0dfc194e313ce8f77070660c208a88d1e30784c8ad913b0e970c94153": rpc error: code = 2 desc
 = failed to delete container k8s_POD_nginx-service-103607745-53qlv_default_d4ab2fd8-9cc4-11e7-9249-525400123456_0 in pod sandbox 19a753f0dfc194e313ce8f77070660c208a88d1e30784c8ad913b0e970c94153: `/usr/local/bin
/cc-runtime delete --force 19a753f0dfc194e313ce8f77070660c208a88d1e30784c8ad913b0e970c94153` failed: No multicast available for CTL channel
Sep 18 22:58:49 singlevm kubelet[28005]:   (exit status 1)

cc-runtime log also shows the same error, with no further information:

2017-09-18 22:58:48.180234981 +0000 UTC:31855:cc-runtime:info:Stopping shim PID 31628
2017-09-18 22:58:48.204010604 +0000 UTC:31855:cc-runtime:info:cgroup path /sys/fs/cgroup/cpu mounted
2017-09-18 22:58:48.22810537 +0000 UTC:31855:cc-runtime:info:cgroup path /sys/fs/cgroup/pids mounted
2017-09-18 22:58:48.512236462 +0000 UTC:31859:cc-runtime:info:cc-runtime (version 3.0.0-beta.2, commit f8f8c6ca784b19b52ba2c5fb0e3a145da264d463) called as: [delete --force 69bfba13db090b6db6c54faf372240d0d3fe2fc
6f5b73d2f0a4c42bc04158ede]
2017-09-18 22:58:48.537868943 +0000 UTC:31859:cc-runtime:info:Using configuration file "/etc/clear-containers/configuration.toml"
2017-09-18 22:58:48.623795326 +0000 UTC:31859:cc-runtime:info:Stopping shim PID 31438
2017-09-18 22:58:48.650099483 +0000 UTC:31859:cc-runtime:info:cgroup path /sys/fs/cgroup/cpu mounted
2017-09-18 22:58:48.672832507 +0000 UTC:31859:cc-runtime:info:cgroup path /sys/fs/cgroup/pids mounted
2017-09-18 22:58:48.869804035 +0000 UTC:31874:cc-runtime:info:cc-runtime (version 3.0.0-beta.2, commit f8f8c6ca784b19b52ba2c5fb0e3a145da264d463) called as: [delete --force 5b985f7f02bdfd2a14074536c1f78f120c3fe56
018a704e75367f388e3a9c72e]
2017-09-18 22:58:48.893521952 +0000 UTC:31874:cc-runtime:info:Using configuration file "/etc/clear-containers/configuration.toml"
2017-09-18 22:58:48.97881162 +0000 UTC:31874:cc-runtime:info:Stopping shim PID 31343
2017-09-18 22:58:49.009998845 +0000 UTC:31874:cc-runtime:info:cgroup path /sys/fs/cgroup/cpu mounted
2017-09-18 22:58:49.033277483 +0000 UTC:31874:cc-runtime:info:cgroup path /sys/fs/cgroup/pids mounted
2017-09-18 22:58:49.194908701 +0000 UTC:31879:cc-runtime:info:cc-runtime (version 3.0.0-beta.2, commit f8f8c6ca784b19b52ba2c5fb0e3a145da264d463) called as: [delete --force 4a5210c5f5c2fb30a372d3206b57843f880fc74
8d71cd8a685585f64ff411628]
2017-09-18 22:58:49.217383324 +0000 UTC:31879:cc-runtime:info:Using configuration file "/etc/clear-containers/configuration.toml"
2017-09-18 22:58:49.28467549 +0000 UTC:31879:cc-runtime:error:No multicast available for CTL channel
2017-09-18 22:58:49.345721773 +0000 UTC:31883:cc-runtime:info:cc-runtime (version 3.0.0-beta.2, commit f8f8c6ca784b19b52ba2c5fb0e3a145da264d463) called as: [delete --force 19a753f0dfc194e313ce8f77070660c208a88d1
e30784c8ad913b0e970c94153]
2017-09-18 22:58:49.373578653 +0000 UTC:31883:cc-runtime:info:Using configuration file "/etc/clear-containers/configuration.toml"
2017-09-18 22:58:49.437917912 +0000 UTC:31883:cc-runtime:error:No multicast available for CTL channel
2017-09-18 22:58:52.717491742 +0000 UTC:31889:cc-runtime:info:cc-runtime (version 3.0.0-beta.2, commit f8f8c6ca784b19b52ba2c5fb0e3a145da264d463) called as: [list]
2017-09-18 22:58:52.74059839 +0000 UTC:31889:cc-runtime:info:Using configuration file "/etc/clear-containers/configuration.toml"

[jamiehannaford]

@jodh-intel @bergwolf Have you seen this before? Also, should this be ported over to https://github.com/kata-containers/runtime?

[chavafg]

@jamiehannaford I opened https://github.com/kata-containers/runtime/issues/197 which is similar to this one.