bugs
bugs copied to clipboard
Unable to run kind node in k8s pod on CoreOS
Issue Report
Bug
I am currently trying to run a kind node inside a Kubernetes Pod running on a CoreOS Node. The Pod manifest looks like this:
apiVersion: v1
kind: Pod
metadata:
name: kind-worker-1
namespace: kind
labels:
app: kind-node
node: worker-1
spec:
containers:
- env:
- name: PATH
value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
- name: container
value: docker
image: kindest/node:v1.16.3
name: kind-node
resources:
limits:
cpu: "2"
memory: 8Gi
requests:
cpu: "2"
memory: 8Gi
securityContext:
privileged: true
stdin: true
volumeMounts:
- mountPath: /lib/modules
name: modules
readOnly: true
- mountPath: /sys/fs/cgroup
name: cgroup
- mountPath: /var/lib/docker
name: dind-storage
volumes:
- emptyDir: {}
name: dind-storage
- hostPath:
path: /lib/modules
type: Directory
name: modules
- hostPath:
path: /sys/fs/cgroup
type: Directory
name: cgroup
I am able to bootstrap the node, join the cluster and so on...
But about every 50 seconds, the systemd services for kubelet
and containerd
inside the Pod are restarted after messages from the kubelet looking like this:
Dec 17 12:14:07 kind-worker-1 kubelet[2903]: I1217 12:14:07.676823 2903 pod_container_manager_linux.go:166] Attempt to kill process with pid: 1
Dec 17 12:14:07 kind-worker-1 kubelet[2903]: I1217 12:14:07.676850 2903 pod_container_manager_linux.go:166] Attempt to kill process with pid: 2896
Dec 17 12:14:07 kind-worker-1 kubelet[2903]: I1217 12:14:07.676861 2903 pod_container_manager_linux.go:166] Attempt to kill process with pid: 2903
Dec 17 12:14:08 kind-worker-1 systemd[1]: kubelet.service: Service RestartSec=1s expired, scheduling restart.
Dec 17 12:14:08 kind-worker-1 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 70.
Dec 17 12:14:08 kind-worker-1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Dec 17 12:14:08 kind-worker-1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Therefore the Node can't stay ready and can't run any Pods.
Container Linux Version
$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=2135.6.0
VERSION_ID=2135.6.0
BUILD_ID=2019-07-30-0722
PRETTY_NAME="Container Linux by CoreOS 2135.6.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
Environment
What hardware/cloud provider/hypervisor is being used to run Container Linux? aws
Expected Behavior
systemd services should keep running inside a docker container.
Actual Behavior
systemd services are killed about every 50s.
Reproduction Steps
- run a Pod with a
kindest/node
image on a CoreOS Node - try to bootstrap a kind cluster
Other Information
I already read through the issues over at kind (mainly https://github.com/kubernetes-sigs/kind/issues/303 and https://github.com/kubernetes-sigs/kind/issues/890) and tried all the suggestions, but none of them seem to fix the problem on CoreOS.
/cc @BenTheElder /cc @aojea
Thanks for the report.
I'm a bit confused here as I don't think we ship any kubelet.service
with Container Linux.
Where is that unit running, on the host or within a pod? And how does it look like?
Sorry for the confusion.
So the kubelet service is actually running inside the kindest/node
container after joining the cluster via kubeadm
.
The unit is generated by kubeadm and looks something like this:
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
This and also the containerd service inside the kind container are killed/crashing every 50s and I am not sure what to look for here...
hi @lucab (kind developer here), nice to meet you again :) Kind is running Kubernetes in Docker, in this case Tim is trying to run Kubernetes in Docker in Kubernetes and I was wondering if CoreOS has some security or filesystems restrictions that can be causing this behavior.
Thanks, for clarifying @aojea
So, if I understood correctly, there is a dedicated systemd session running in a pod, and the kubelet.service
running below it.
I would probably have a look at journalctl
and systemctl status
for those units, which should tell you at least why each unit is restarting.
Other than that, the logs in the original ticket show something on a killing spree trying to kill a bunch of things, including pid-1, which does not seem a very healthy thing to do.
Yes, exactly. That is the problem, which I am currently stuck at.
I attached some journal logs from the kubelet
and containerd
services shortly before they are killed.
containerd.log
kubelet.log
Here is some output from systemctl status
shortly before and after the services are killed:
root@kind-worker-1:/# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/kind/systemd/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Wed 2019-12-18 11:52:58 UTC; 21s ago
Docs: http://kubernetes.io/docs/
Main PID: 14026 (kubelet)
Tasks: 18 (limit: 4915)
Memory: 33.0M
CGroup: /kubepods/podfb87cd2a-bc2d-46ac-b5c0-e1673b21a438/cce19d47d2587cc2c3996a0d1042d70ef25434abe454edd5a269e98edaae670d/system.slice/kubelet.service
└─14026 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --fail-swap-
on=false --node-ip=10.241.132.21 --v=7 --fail-swap-on=false
Dec 18 11:53:18 kind-worker-1 kubelet[14026]: I1218 11:53:18.839886 14026 helpers.go:781] eviction manager: observations: signal=imagefs.available, available: 42092580Ki, capacity: 48375392Ki, time: 2019-12-18 11:53:18.137623173 +0000 UTC
Dec 18 11:53:18 kind-worker-1 kubelet[14026]: I1218 11:53:18.839911 14026 helpers.go:781] eviction manager: observations: signal=imagefs.inodesFree, available: 12371658, capacity: 12444032, time: 2019-12-18 11:53:18.137623173 +0000 UTC
Dec 18 11:53:18 kind-worker-1 kubelet[14026]: I1218 11:53:18.839922 14026 helpers.go:781] eviction manager: observations: signal=pid.available, available: 31943, capacity: 32Ki, time: 2019-12-18 11:53:18.838649371 +0000 UTC m=+20.823194956
Dec 18 11:53:18 kind-worker-1 kubelet[14026]: I1218 11:53:18.839932 14026 helpers.go:781] eviction manager: observations: signal=memory.available, available: 13818732Ki, capacity: 15950572Ki, time: 2019-12-18 11:53:18.809029308 +0000 UTC m=+20.793574907
Dec 18 11:53:18 kind-worker-1 kubelet[14026]: I1218 11:53:18.839942 14026 helpers.go:781] eviction manager: observations: signal=allocatableMemory.available, available: 15151388Ki, capacity: 15950572Ki, time: 2019-12-18 11:53:18.839809486 +0000 UTC m=+20.824355082
Dec 18 11:53:18 kind-worker-1 kubelet[14026]: I1218 11:53:18.839951 14026 helpers.go:781] eviction manager: observations: signal=nodefs.available, available: 42092580Ki, capacity: 48375392Ki, time: 2019-12-18 11:53:18.809029308 +0000 UTC m=+20.793574907
Dec 18 11:53:18 kind-worker-1 kubelet[14026]: I1218 11:53:18.839961 14026 helpers.go:781] eviction manager: observations: signal=nodefs.inodesFree, available: 12371658, capacity: 12444032, time: 2019-12-18 11:53:18.809029308 +0000 UTC m=+20.793574907
Dec 18 11:53:18 kind-worker-1 kubelet[14026]: I1218 11:53:18.839984 14026 eviction_manager.go:320] eviction manager: no resources are starved
Dec 18 11:53:18 kind-worker-1 kubelet[14026]: I1218 11:53:18.909515 14026 config.go:100] Looking for [api file], have seen map[]
Dec 18 11:53:19 kind-worker-1 kubelet[14026]: I1218 11:53:19.009526 14026 config.go:100] Looking for [api file], have seen map[]
root@kind-worker-1:/# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/kind/systemd/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: signal) since Wed 2019-12-18 11:53:20 UTC; 66ms ago
Docs: http://kubernetes.io/docs/
Process: 14026 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=killed, signal=KILL)
Main PID: 14026 (code=killed, signal=KILL)
root@kind-worker-1:/# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/kind/systemd/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Wed 2019-12-18 11:53:22 UTC; 80ms ago
Docs: http://kubernetes.io/docs/
Main PID: 14561 (kubelet)
Tasks: 12 (limit: 4915)
Memory: 15.0M
CGroup: /kubepods/podfb87cd2a-bc2d-46ac-b5c0-e1673b21a438/cce19d47d2587cc2c3996a0d1042d70ef25434abe454edd5a269e98edaae670d/system.slice/kubelet.service
└─14561 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --fail-swap-
on=false --node-ip=10.241.132.21 --v=7 --fail-swap-on=false
Dec 18 11:53:22 kind-worker-1 kubelet[14561]: I1218 11:53:22.089609 14561 config.go:412] Receiving a new pod "blackbox-exporter-855496dd6b-lw729_kube-system(2470f68d-1cb0-447a-8c20-d28d0a39b53a)"
Dec 18 11:53:22 kind-worker-1 kubelet[14561]: I1218 11:53:22.089665 14561 config.go:412] Receiving a new pod "coredns-7cbffb9d6b-678cz_kube-system(5349c653-4bd1-427e-9c6e-fbc93053b798)"
Dec 18 11:53:22 kind-worker-1 kubelet[14561]: I1218 11:53:22.089675 14561 config.go:412] Receiving a new pod "coredns-7cbffb9d6b-rrgr9_kube-system(ec31a47c-4b3d-4c70-939a-e82080211d12)"
Dec 18 11:53:22 kind-worker-1 kubelet[14561]: I1218 11:53:22.089685 14561 config.go:412] Receiving a new pod "metrics-server-7c8756b968-8mn6l_kube-system(df37faad-e7ed-463a-862b-595961f0f2a8)"
Dec 18 11:53:22 kind-worker-1 kubelet[14561]: I1218 11:53:22.089695 14561 config.go:412] Receiving a new pod "kube-proxy-6lzmh_kube-system(a7e622f1-ec41-4a9c-8ded-0d13785875a3)"
Dec 18 11:53:22 kind-worker-1 kubelet[14561]: I1218 11:53:22.089704 14561 config.go:412] Receiving a new pod "node-exporter-fpg7g_kube-system(3c7b8c0f-0ee0-41ad-a68f-b9f5a6c37cbf)"
Dec 18 11:53:22 kind-worker-1 kubelet[14561]: I1218 11:53:22.089714 14561 config.go:412] Receiving a new pod "vpn-shoot-6746d498ff-n59pt_kube-system(9ea15479-1211-4760-9927-2fa34d5071bd)"
Dec 18 11:53:22 kind-worker-1 kubelet[14561]: I1218 11:53:22.089723 14561 config.go:412] Receiving a new pod "node-problem-detector-trj4n_kube-system(a1e0c1d4-89b2-411c-9d72-212574432069)"
Dec 18 11:53:22 kind-worker-1 kubelet[14561]: I1218 11:53:22.090770 14561 round_trippers.go:446] Response Status: 200 OK in 1 milliseconds
Dec 18 11:53:22 kind-worker-1 kubelet[14561]: I1218 11:53:22.090812 14561 round_trippers.go:446] Response Status: 200 OK in 2 milliseconds
root@kind-worker-1:/# systemctl status containerd
● containerd.service - containerd container runtime
Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-12-18 11:56:34 UTC; 46s ago
Docs: https://containerd.io
Process: 20885 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 20886 (containerd)
Tasks: 105
Memory: 102.0M
CGroup: /kubepods/podfb87cd2a-bc2d-46ac-b5c0-e1673b21a438/cce19d47d2587cc2c3996a0d1042d70ef25434abe454edd5a269e98edaae670d/system.slice/containerd.service
├─20886 /usr/local/bin/containerd
├─21694 /usr/local/bin/containerd-shim-runc-v1 -namespace k8s.io -id fb1d6d317be1643a4c86ad16e7ccdb5be807228729ab9cdc49c41866ab6be26e -address /run/containerd/containerd.sock
├─21991 /usr/local/bin/containerd-shim-runc-v1 -namespace k8s.io -id d11d6c948efcd0e49090c3ecaade330d4d2c52ffb60322649e8da84bd408f15f -address /run/containerd/containerd.sock
├─22036 runc --root /run/containerd/runc/k8s.io --log /run/containerd/io.containerd.runtime.v2.task/k8s.io/d11d6c948efcd0e49090c3ecaade330d4d2c52ffb60322649e8da84bd408f15f/log.json --log-format json create --bundle /run/containerd/io.containerd.runtime.v2.task/k8s.io/d11d6c
948efcd0e49090c3ecaade330d4d2c52ffb60322649e8da84bd408f15f --pid-file /run/containerd/io.containerd.runtime.v2.task/k8s.io/d11d6c948efcd0e49090c3ecaade330d4d2c52ffb60322649e8da84bd408f15f/init.pid d11d6c948efcd0e49090c3ecaade330d4d2c52ffb60322649e8da84bd408f15f
├─22066 /usr/local/bin/containerd-shim-runc-v1 -namespace k8s.io -id 3fdb961df304f58c4f9ae0ebd6613b9a3b4bf9f8b2eb7a659ecfd776a9bb7a02 -address /run/containerd/containerd.sock
├─22080 /usr/local/bin/containerd-shim-runc-v1 -namespace k8s.io -id e1444dfdb2e3a399e44b737dc6d4c8b6e4519287ff4d16e56da683e3cb3383d5 -address /run/containerd/containerd.sock
├─22089 runc --root /run/containerd/runc/k8s.io --log /run/containerd/io.containerd.runtime.v2.task/k8s.io/3fdb961df304f58c4f9ae0ebd6613b9a3b4bf9f8b2eb7a659ecfd776a9bb7a02/log.json --log-format json create --bundle /run/containerd/io.containerd.runtime.v2.task/k8s.io/3fdb96
1df304f58c4f9ae0ebd6613b9a3b4bf9f8b2eb7a659ecfd776a9bb7a02 --pid-file /run/containerd/io.containerd.runtime.v2.task/k8s.io/3fdb961df304f58c4f9ae0ebd6613b9a3b4bf9f8b2eb7a659ecfd776a9bb7a02/init.pid 3fdb961df304f58c4f9ae0ebd6613b9a3b4bf9f8b2eb7a659ecfd776a9bb7a02
├─22107 /usr/local/bin/containerd-shim-runc-v1 -namespace k8s.io -id ce06d36dddb80a4affeefe37f60cd844f392d5f75102e1e5fd4d378b9abda717 -address /run/containerd/containerd.sock
├─22154 /usr/local/bin/containerd-shim-runc-v1 -namespace k8s.io -id 1a2a2b818d89db7fa2377e20df61b85d60d3fb09b4528d8b4411957b8a04b3f8 -address /run/containerd/containerd.sock
└─22274 /usr/local/bin/containerd-shim-runc-v1 -namespace k8s.io -id 8ec35bb8944b47df5cabd8e69f57670ee193fa21490f30d6c2e88d315f527338 -address /run/containerd/containerd.sock
Dec 18 11:57:19 kind-worker-1 containerd[20886]: time="2019-12-18T11:57:19.527979778Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/e1444dfdb2e3a399e44b737dc6d4c8b6e4519287ff4d16e56da683e3cb3383d5 pid=22080
Dec 18 11:57:19 kind-worker-1 containerd[20886]: time="2019-12-18T11:57:19.555670665Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/ce06d36dddb80a4affeefe37f60cd844f392d5f75102e1e5fd4d378b9abda717 pid=22107
Dec 18 11:57:19 kind-worker-1 containerd[20886]: time="2019-12-18T11:57:19.584758374Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/1a2a2b818d89db7fa2377e20df61b85d60d3fb09b4528d8b4411957b8a04b3f8 pid=22154
Dec 18 11:57:19 kind-worker-1 containerd[20886]: time="2019-12-18T11:57:19.687899264Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-7cbffb9d6b-rrgr9,Uid:ec31a47c-4b3d-4c70-939a-e82080211d12,Namespace:kube-system,Attempt:0,} returns sandbox id \"ce06d36ddd
b80a4affeefe37f60cd844f392d5f75102e1e5fd4d378b9abda717\""
Dec 18 11:57:19 kind-worker-1 containerd[20886]: time="2019-12-18T11:57:19.690334540Z" level=info msg="PullImage \"coredns/coredns:1.6.3\""
Dec 18 11:57:19 kind-worker-1 containerd[20886]: time="2019-12-18T11:57:19.698089495Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:metrics-server-7c8756b968-8mn6l,Uid:df37faad-e7ed-463a-862b-595961f0f2a8,Namespace:kube-system,Attempt:3,} returns sandbox id \"1a2
a2b818d89db7fa2377e20df61b85d60d3fb09b4528d8b4411957b8a04b3f8\""
Dec 18 11:57:19 kind-worker-1 containerd[20886]: time="2019-12-18T11:57:19.700981925Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-7cbffb9d6b-678cz,Uid:5349c653-4bd1-427e-9c6e-fbc93053b798,Namespace:kube-system,Attempt:0,} returns sandbox id \"e1444dfdb2
e3a399e44b737dc6d4c8b6e4519287ff4d16e56da683e3cb3383d5\""
Dec 18 11:57:20 kind-worker-1 containerd[20886]: time="2019-12-18T11:57:20.038419926Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:node-problem-detector-trj4n,Uid:a1e0c1d4-89b2-411c-9d72-212574432069,Namespace:kube-system,Attempt:0,}"
Dec 18 11:57:20 kind-worker-1 containerd[20886]: time="2019-12-18T11:57:20.086019519Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/8ec35bb8944b47df5cabd8e69f57670ee193fa21490f30d6c2e88d315f527338 pid=22274
Dec 18 11:57:20 kind-worker-1 containerd[20886]: time="2019-12-18T11:57:20.147494504Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:node-problem-detector-trj4n,Uid:a1e0c1d4-89b2-411c-9d72-212574432069,Namespace:kube-system,Attempt:0,} returns sandbox id \"8ec35bb
8944b47df5cabd8e69f57670ee193fa21490f30d6c2e88d315f527338\""
root@kind-worker-1:/# systemctl status containerd
● containerd.service - containerd container runtime
Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: signal) since Wed 2019-12-18 11:57:20 UTC; 703ms ago
Docs: https://containerd.io
Process: 20885 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 20886 ExecStart=/usr/local/bin/containerd (code=killed, signal=KILL)
Main PID: 20886 (code=killed, signal=KILL)
Tasks: 0
Memory: 4.1M
CGroup: /kubepods/podfb87cd2a-bc2d-46ac-b5c0-e1673b21a438/cce19d47d2587cc2c3996a0d1042d70ef25434abe454edd5a269e98edaae670d/system.slice/containerd.service
root@kind-worker-1:/# systemctl status containerd
● containerd.service - containerd container runtime
Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-12-18 11:57:22 UTC; 1s ago
Docs: https://containerd.io
Process: 22510 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 22516 (containerd)
Tasks: 14
Memory: 29.7M
CGroup: /kubepods/podfb87cd2a-bc2d-46ac-b5c0-e1673b21a438/cce19d47d2587cc2c3996a0d1042d70ef25434abe454edd5a269e98edaae670d/system.slice/containerd.service
└─22516 /usr/local/bin/containerd
Dec 18 11:57:22 kind-worker-1 containerd[22516]: time="2019-12-18T11:57:22.101516635Z" level=info msg="Get image filesystem path \"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
Dec 18 11:57:22 kind-worker-1 containerd[22516]: time="2019-12-18T11:57:22.101961881Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Dec 18 11:57:22 kind-worker-1 containerd[22516]: time="2019-12-18T11:57:22.102229859Z" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Dec 18 11:57:22 kind-worker-1 containerd[22516]: time="2019-12-18T11:57:22.102297986Z" level=info msg=serving... address=/run/containerd/containerd.sock
Dec 18 11:57:22 kind-worker-1 containerd[22516]: time="2019-12-18T11:57:22.102312737Z" level=info msg="containerd successfully booted in 0.054324s"
Dec 18 11:57:22 kind-worker-1 containerd[22516]: time="2019-12-18T11:57:22.109845987Z" level=info msg="Start subscribing containerd event"
Dec 18 11:57:22 kind-worker-1 containerd[22516]: time="2019-12-18T11:57:22.109929056Z" level=info msg="Start recovering state"
Dec 18 11:57:22 kind-worker-1 containerd[22516]: time="2019-12-18T11:57:22.123202529Z" level=info msg="Start event monitor"
Dec 18 11:57:22 kind-worker-1 containerd[22516]: time="2019-12-18T11:57:22.123240646Z" level=info msg="Start snapshots syncer"
Dec 18 11:57:22 kind-worker-1 containerd[22516]: time="2019-12-18T11:57:22.123250853Z" level=info msg="Start streaming server"