bugs
bugs copied to clipboard
Kubernetes liveness probes with docker exec fail randomly
Issue Report
Bug
Kubernetes liveness probes fail randomly on this version of coreos. There is a bug regarding the used runc version ...
runc --version
runc version 1.0.0-rc5+dev.docker-18.06
commit: a592beb5bc4c4092b1b1bac971afed27687340c5
spec: 1.0.0
See here
user 5m 5m 1 user-sqsworker-55f4f9494f-glnm7.15b76be66de646eb Pod spec.containers{rails} Warning Unhealthy kubelet, ip-172-31-101-183.eu-west-1.compute.internal Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "process_linux.go:90: adding pid 18580 to cgroups caused \"failed to write 18580 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/podb72c8c56-b538-11e9-9f9c-06d6b3e699b6/1075b19a94bb045fdf72bfb0133bbbc721f3e04c43a99ded2d0f5eba6f34e7ca/cgroup.procs: invalid argument\"": unknown
This error happens randomly and we cannot provoke it but as it happens with our cni pods as well, thats why it is a big issue for us.
Container Linux Version
cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=2135.6.0
VERSION_ID=2135.6.0
BUILD_ID=2019-07-30-0722
PRETTY_NAME="Container Linux by CoreOS 2135.6.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
Environment
aws ec2 instance m5.xlarge
Expected Behavior
Liveness probe fails only on error within the pod
Actual Behavior
Liveness probes fails randomly on setting cgroups.procs.
Reproduction Steps
- Use kubernetes 1.11.10 deployed by kops 1.11.1 on aws
- Wait and watch events in all namespaces for this error to occur
Other Information
Thanks for the report. Did this work properly in a previous version of Container Linux?
We use an immutable approach and disable the update-engine.
It started when we migrated to CoreOS-stable-2079.3.0-hvm
and is still with CoreOS-stable-2135.5.0-hvm
that we are using now.
Is there any best practice on how to replace runc on the system properly for testing?
It started when we migrated to
CoreOS-stable-2079.3.0-hvm
Which version were you using before that?
Hi, I had the same problem.
like this:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy <invalid> kubelet, k8s-node47 Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "process_linux.go:90: adding pid 9319 to cgroups caused \"failed to write 9319 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/podb54e8b22-eb09-11e9-a0b0-fa163e623fb1/a76f45c56400a1e956b12d0977b2522cf4856d6744cf2b02075334014cdcb57c/cgroup.procs: invalid argument\"": unknown
kubernetes version:
[root@k8s-node47 ~]# kubelet --version
Kubernetes v1.13.5
CentOS info:
[root@k8s-node47 ~]# uname -a
Linux k8s-node47 4.18.9-1.el7.elrepo.x86_64 #1 SMP Thu Sep 20 09:04:54 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@k8s-node47 ~]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
Docker info:
[root@k8s-node47 ~]# docker info
Containers: 38
Running: 34
Paused: 0
Stopped: 4
Images: 19
Server Version: 18.06.3-ce
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: a592beb5bc4c4092b1b1bac971afed27687340c5
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.18.9-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.66GiB
Name: k8s-node47
ID: PW4G:TDGS:X3N3:5JBB:NOXS:R3F7:5BGF:NX3N:TCAI:PDWQ:WLQH:6ZR6
Docker Root Dir: /data/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Have you found this reason?