bugs
bugs copied to clipboard
Published
20 hours ago •
coreos
coreos
/run/torcx/bin/docker: Error response from daemon: mounting shm tmpfs: no space left on device.
Issue Report
Bug
/run/torcx/bin/docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/7c0178cc475ec5e28183458635a155aaedc53766f345a2018960fdb90e769d23-init/merged: no space left on device. See '/run/torcx/bin/docker run --help'.
Container Linux Version
$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1688.5.3
VERSION_ID=1688.5.3
BUILD_ID=2018-04-03-0547
PRETTY_NAME="Container Linux by CoreOS 1688.5.3 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
k8s-node-12 ~ # df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 13M 32G 1% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda9 460G 184G 258G 42% /
/dev/mapper/usr 985M 704M 231M 76% /usr
none 32G 154M 32G 1% /run/torcx/unpack
/dev/sda1 128M 49M 80M 38% /boot
tmpfs 32G 0 32G 0% /media
tmpfs 32G 0 32G 0% /tmp
/dev/sda6 108M 84K 99M 1% /usr/share/oem
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/1b11d843-43df-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/1b182931-43df-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-mk28p
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/1b13d0bb-43df-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/calico-node-token-ngk9x
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/1b13d0bb-43df-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/etcd-certs
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/ea91f15e66bb8cdfb253328f891dc808862cdd35848b93f00b3ea7cb0af958a0/merged
shm 64M 0 64M 0% /var/lib/docker/containers/1dfea65f206dc2e45d8a7f8a0a6d578d661c3e324d37ee3b8e73977f71744a2a/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/40d00c7368f6c9518687b5e7e6d0359e937e9bef3bd8e742880c362052c45465/merged
shm 64M 0 64M 0% /var/lib/docker/containers/a419d6d6566332473e6e16d410c7b2084cbe3876e446047672ef1ed1d0f83e9c/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/0260560c844914ff84e9f025a5c0622c7116b9db290f75981187340f2ecff34e/merged
shm 64M 0 64M 0% /var/lib/docker/containers/20624599bae6c35c77990fa3d25d0055223d1673dc3b1d26da34612f9a74b2bc/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/c6c038e8e687569ad7c5458305433fdf5416252335d5927defb650a42dedaeab/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/051ee43784851436ec77f012cdac384e53730beb90f9ea73c33e93196a93aee6/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/21de3f3373ea72296a65ccf0a8d085110cc56ef76e009198e728ab5863aac69c/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/4b37f4592ec5229f6f680ce610d1ffebbcb36e3fe34558655f53b2f720e1a891/merged
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/cba199ad-44a0-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-94zbb
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/5d0fbc50f9eeca6534047d3ad27c4a49c484d0629c829487602cd293d5fda382/merged
shm 64M 0 64M 0% /var/lib/docker/containers/f94675cf2abedbea8f8208badd2221290066bbe53bac0594fc4a8fc7095ec549/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/90915e8f884f0b7efefd7d9101d9c0b4e4d4a247dee39a832bb577cc0353e163/merged
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/e8cfdbb7-44ab-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-94zbb
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/c42e92c51ab2e9113f7323c1a2640eda04723fc245b556b13940c9c17c593ec4/merged
shm 64M 0 64M 0% /var/lib/docker/containers/42cf317db095c22d97fdad7035258bd32bbd66df1228d068e88c77728804c865/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/4fe755a97a302a396962f05fce1fd139f2163f2bbd620604e9dc3b7b74985e67/merged
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/6dadc115-4924-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/7fdee19e52cd54df08b545785d0e40dcfecadd7334399c3d8a1482f17503c7f3/merged
shm 64M 0 64M 0% /var/lib/docker/containers/a74e1dc07978726079409f0f3d550874c40fd9b1c7744ac7d42ba9a60fdab041/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/dece13d3154453e887e424aaf08a8d3cdbd9722fb8b6a25c86d811e74acfb991/merged
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/22e381e8-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/7bf74fcd68554778f42e50a22747b0ee31681fe08b8ad25aaf5a6df7d6e78e8b/merged
shm 64M 2.1M 62M 4% /var/lib/docker/containers/970895682bb384cd461c5c9cb363ebf2a439b95de5e9b721a1897b204ea0539a/shm
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/37c9da71-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/b0a65779be99fba0035a5ddd8f4226192185570b0ab6b63c3f62b6bd1e542261/merged
shm 64M 2.1M 62M 4% /var/lib/docker/containers/85fd0403ff40255b601d98a09c601934747bfb1207a9bcc4b01cbf3c90b1cd0b/shm
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/a887279b-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/6ad605b2e757a7cbc3d3941b3d9769fc1215968b629f6c539723856a52efe149/merged
shm 64M 12K 64M 1% /var/lib/docker/containers/a7bd2ca43787bcb3cfaf3aaeebc6ef7d234ceddc53a5442126b5a5bdc54c7bae/shm
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/a9654511-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/52b9da8952d690b077ddd788045c7e6683088146ee2de5ce17bffb55959c506e/merged
shm 64M 12K 64M 1% /var/lib/docker/containers/483d8bb8e811467a9c3ac745b6867ba16fb05a6461eef6489c3c1a3065dfe929/shm
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/aa6f9f9b-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/838f83df99212ec9d4f362c6246f582f1bc070a2c32eaac5d9a971aa81a8e33c/merged
shm 64M 12K 64M 1% /var/lib/docker/containers/f485abcac9e9d468cce9ff8203892baf2ad578886b672a644c62b0218717b938/shm
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/ac7bffc5-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/68f950a5afe35bff7403e90114080c0e89c93843a227d2faefe5dbabb9240c0a/merged
shm 64M 12K 64M 1% /var/lib/docker/containers/906381fd7cc50d94b323fdec99a4f69d74ab829494292f14add69e713f2292e7/shm
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/10a45c46-4a2b-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/76b88cf116a920e4c6512e9880df100d1ee2a94980879bd2bffd967a4fb95b8e/merged
shm 64M 12K 64M 1% /var/lib/docker/containers/2289f61a0ea5776539bc7f0cb5d615ab9bb9ea6ebfc1aaeb55937a4690835c75/shm
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/1ff1ccba-4a2b-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/8c68d5a9bafaad34513cda8307f1891700e23d5ff00ae616d1b23a89a8ad966b/merged
shm 64M 12K 64M 1% /var/lib/docker/containers/edd2bb34d731a7975dd37ba8b1579fc91368fb4b50ddabb6bc464c68a089ede7/shm
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/b53e6505-4a3a-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/b77ba2cfb16a383b816eb79e91de7b8686275060078316fe678a8cc549ca9f41/merged
shm 64M 0 64M 0% /var/lib/docker/containers/e5e2cb318cf4063fb383d7acb22cd3443bfe9f5c60d92ae549f575dba24efee2/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/b3161b581184486a5e8b2e547cd603d7077fef7d5584cdedab13bd3bc2a24602/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/8f929839b62c2e46056917c51b05a0d93dabd6149c197bd1f5c911db4792584e/merged
shm 64M 0 64M 0% /var/lib/docker/containers/78c0c35fe5833d18b6d6d16ae81bd146910755bed831eb09adea3185796f8ff0/shm
overlay 460G 184G 258G 42% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs
overlay 460G 184G 258G 42% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs
tmpfs 32G 0 32G 0% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs/tmp
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/3bb5c6dbe8f7f46bb52b51fad949724c61417b51eeb12f68454067c296a481e1/merged
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/3d0587bd-4f6b-11e8-be69-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/3d1addd0-4f6b-11e8-be69-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/85ee038bfba77b3a046044b3b94fd2f789f0cec93e6e5414bc5913d09f03e3ff/merged
shm 64M 12K 64M 1% /var/lib/docker/containers/d7736b64f084a4e61622247eef9b082df215cad901080d093f086a4656fd61e2/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/10fd5075134265a5367aabceb8aa1ffb46b45844d32092e6a56c102c41affd1b/merged
shm 64M 12K 64M 1% /var/lib/docker/containers/2cf3a0373feb079701ba0c9cb003b2639e5447a6195954487a4d42fd7ab20c58/shm
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/0e55d061-4f73-11e8-be69-ce7be4be3aff/volumes/kubernetes.io~secret/kube-dns-token-66chm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/43032198af411e129e44889ceaa08824a01042fc1e1f11b62bbdc4ecdad5b6ae/merged
shm 64M 0 64M 0% /var/lib/docker/containers/fb0b8c81c0f6d3096f6f22dbc5a377e77b2e0dd28248f02d59d39b058d2f377c/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/43c4898ce4dea7e0a4670623da005ee215edbc9274d84a9ced1ccb8b581fa675/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/2ca7d653e8d622e1f58e55d3d27aa4e0adcef21d1152faa0a6108fc7eaf1e9b6/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/52af42a85e8c6e2d2a5e836bf684641c902f482d6fe81f5f2e9c797180701884/merged
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/b1d66af4-4f79-11e8-be69-ce7be4be3aff/volumes/kubernetes.io~secret/kube-dns-token-66chm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/7a8f6d65ec9583e368a8971dd4a512357bde7b0be7ff4f65fb11ab88e18ce527/merged
shm 64M 0 64M 0% /var/lib/docker/containers/ab5334cb4b295dbd161dc50aee32c5f301b6fa8329516b2e3b4ad0d8c94a14e7/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/962e030855e7493d5d8293900833ab1bcce18a7e90cc0cbc32ddbb99a134fcb0/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/f3633bc895a5433a6bc8599373a003a142bcc5a4a5d291d50e1381f66fe173f6/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/209ba4f77e2794e9373c46890596bbfef140232661f8144577ea2ab8db4f52e2/merged
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/50cb323a-5028-11e8-be69-ce7be4be3aff/volumes/kubernetes.io~secret/kube-dns-token-66chm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/b82c938c6f5387e7d0020a1707abcaf2b190649a992afb934af21534b0c0cdd5/merged
shm 64M 0 64M 0% /var/lib/docker/containers/71dd46e2f91605ca2854c2f812fa0117618126e86b3899ed41ee0a223cdd36f4/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/30aa11ade19c0d3edb016f932656217ada3d624827d938109afcee00927cf50e/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/41ca0c54012a9fccd140d4ffe7f8106e77a582f838af4d1aa11db5d191231f2e/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/378675550d7d737f2fce2a4cc8bcdb5dc41488f360babd0e282abd32e488af5d/merged
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/edebc31d-51cf-11e8-8636-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-74f28
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/75eb35b761239195b7db7af509788ff4452f48357e2bb297db7d3badb066b129/merged
shm 64M 4.0K 64M 1% /var/lib/docker/containers/c38bbf473810b5cf2a6dcf8b21d188b5cd9cf97119d1aa91903eee04aa773d3e/shm
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/f21f8584a02b60099827c609b1a1022e120ac98cca2c2969c81c62ad1802356e/merged
overlay 460G 184G 258G 42% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs/var/lib/docker/overlay2/de41fbe99da802b1167973c93fff008ea5ae661d6a01cf596b861241b5a93ec8/merged
overlay 460G 184G 258G 42% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs/var/lib/docker/overlay2/bf67f1130ccb20579b5f7c50a1d9f0d3459dae26fe2db8ad5923963de3ad75fb/merged
overlay 460G 184G 258G 42% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs/var/lib/docker/overlay2/bf67f1130ccb20579b5f7c50a1d9f0d3459dae26fe2db8ad5923963de3ad75fb/merged/root-disk/var/lib/docker/overlay2/bf67f1130ccb20579b5f7c50a1d9f0d3459dae26fe2db8ad5923963de3ad75fb/merged
overlay 460G 184G 258G 42% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs/var/lib/docker/overlay2/3f86e17b428a9bf2b4be02980a461e0ab0dd27476553c84ea5b3279d551b3f4f/merged
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/e1272c79-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/splash-token-d445r
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/e1480a2a-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/splash-token-d445r
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/9ddb2c64ab25cd2f8688b8fb9dd76287f78f45533df407a8e666af4b5f61a62f/merged
shm 64M 0 64M 0% /var/lib/docker/containers/e50f6e37a253e66699852c0e72ae4ae6eb7d741c67f1997388afe15de3e3b6b7/shm
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/e1d1fc17-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/splash-token-d445r
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/e233b19b-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/splash-token-d445r
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/e317c225-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/splash-token-d445r
tmpfs 32G 12K 32G 1% /var/lib/kubelet/pods/e28ab020-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/splash-token-d445r
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/398fba192b2bdd02d7cce4d5c814a5369a81c805ac5d7b5dfb1f31f63564e346/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/09f0f2f26259087ec851c8e58f8f393b015e6d68b6eec09bc4f3f2084154a472/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/8981515939c853d28776631a9b4f251cc5086c3e9b5da775b606969cb189090e/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/74490bfdb71df7137b594d981dc2555602a4b9a5782c6218a74614993d0ac2da/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/9bc17139cac05e81a26c184a15b51722189dbcb10bdb03c618eb0bd6a50407b5/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/9183a101d103a2ef358185fde5a7ba4503b4c5a4e23fb791d2c015ad8cf63c19/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/cb885310d7f9bbe8090307135aee1e659b3ceee274061f49ed4f82b32d2db319/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/f35478fdb7882e70f5dca3c873c3d016d53047840004f83d91866b013a83383b/merged
overlay 460G 184G 258G 42% /var/lib/docker/overlay2/590b971b3bda3c8e9b3176e066a7200c77e560e9cab9c857cd8ba90d8f64d5fa/merged
k8s-node-12 ~ # df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 7.9M 355 7.9M 1% /dev
tmpfs 7.9M 1 7.9M 1% /dev/shm
tmpfs 7.9M 1.2K 7.9M 1% /run
tmpfs 7.9M 16 7.9M 1% /sys/fs/cgroup
/dev/sda9 118M 301K 118M 1% /
/dev/mapper/usr 254K 20K 235K 8% /usr
none 7.9M 41 7.9M 1% /run/torcx/unpack
/dev/sda1 0 0 0 - /boot
tmpfs 7.9M 1 7.9M 1% /media
tmpfs 7.9M 6 7.9M 1% /tmp
/dev/sda6 32K 11 32K 1% /usr/share/oem
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/1b11d843-43df-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/1b182931-43df-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-mk28p
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/1b13d0bb-43df-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/calico-node-token-ngk9x
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/1b13d0bb-43df-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/etcd-certs
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/ea91f15e66bb8cdfb253328f891dc808862cdd35848b93f00b3ea7cb0af958a0/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/1dfea65f206dc2e45d8a7f8a0a6d578d661c3e324d37ee3b8e73977f71744a2a/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/40d00c7368f6c9518687b5e7e6d0359e937e9bef3bd8e742880c362052c45465/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/a419d6d6566332473e6e16d410c7b2084cbe3876e446047672ef1ed1d0f83e9c/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/0260560c844914ff84e9f025a5c0622c7116b9db290f75981187340f2ecff34e/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/20624599bae6c35c77990fa3d25d0055223d1673dc3b1d26da34612f9a74b2bc/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/c6c038e8e687569ad7c5458305433fdf5416252335d5927defb650a42dedaeab/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/051ee43784851436ec77f012cdac384e53730beb90f9ea73c33e93196a93aee6/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/21de3f3373ea72296a65ccf0a8d085110cc56ef76e009198e728ab5863aac69c/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/4b37f4592ec5229f6f680ce610d1ffebbcb36e3fe34558655f53b2f720e1a891/merged
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/cba199ad-44a0-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-94zbb
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/5d0fbc50f9eeca6534047d3ad27c4a49c484d0629c829487602cd293d5fda382/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/f94675cf2abedbea8f8208badd2221290066bbe53bac0594fc4a8fc7095ec549/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/90915e8f884f0b7efefd7d9101d9c0b4e4d4a247dee39a832bb577cc0353e163/merged
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/e8cfdbb7-44ab-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-94zbb
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/c42e92c51ab2e9113f7323c1a2640eda04723fc245b556b13940c9c17c593ec4/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/42cf317db095c22d97fdad7035258bd32bbd66df1228d068e88c77728804c865/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/4fe755a97a302a396962f05fce1fd139f2163f2bbd620604e9dc3b7b74985e67/merged
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/6dadc115-4924-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/7fdee19e52cd54df08b545785d0e40dcfecadd7334399c3d8a1482f17503c7f3/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/a74e1dc07978726079409f0f3d550874c40fd9b1c7744ac7d42ba9a60fdab041/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/dece13d3154453e887e424aaf08a8d3cdbd9722fb8b6a25c86d811e74acfb991/merged
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/22e381e8-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/7bf74fcd68554778f42e50a22747b0ee31681fe08b8ad25aaf5a6df7d6e78e8b/merged
shm 7.9M 5 7.9M 1% /var/lib/docker/containers/970895682bb384cd461c5c9cb363ebf2a439b95de5e9b721a1897b204ea0539a/shm
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/37c9da71-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/b0a65779be99fba0035a5ddd8f4226192185570b0ab6b63c3f62b6bd1e542261/merged
shm 7.9M 5 7.9M 1% /var/lib/docker/containers/85fd0403ff40255b601d98a09c601934747bfb1207a9bcc4b01cbf3c90b1cd0b/shm
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/a887279b-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/6ad605b2e757a7cbc3d3941b3d9769fc1215968b629f6c539723856a52efe149/merged
shm 7.9M 4 7.9M 1% /var/lib/docker/containers/a7bd2ca43787bcb3cfaf3aaeebc6ef7d234ceddc53a5442126b5a5bdc54c7bae/shm
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/a9654511-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/52b9da8952d690b077ddd788045c7e6683088146ee2de5ce17bffb55959c506e/merged
shm 7.9M 4 7.9M 1% /var/lib/docker/containers/483d8bb8e811467a9c3ac745b6867ba16fb05a6461eef6489c3c1a3065dfe929/shm
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/aa6f9f9b-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/838f83df99212ec9d4f362c6246f582f1bc070a2c32eaac5d9a971aa81a8e33c/merged
shm 7.9M 4 7.9M 1% /var/lib/docker/containers/f485abcac9e9d468cce9ff8203892baf2ad578886b672a644c62b0218717b938/shm
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/ac7bffc5-4a14-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/68f950a5afe35bff7403e90114080c0e89c93843a227d2faefe5dbabb9240c0a/merged
shm 7.9M 4 7.9M 1% /var/lib/docker/containers/906381fd7cc50d94b323fdec99a4f69d74ab829494292f14add69e713f2292e7/shm
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/10a45c46-4a2b-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/76b88cf116a920e4c6512e9880df100d1ee2a94980879bd2bffd967a4fb95b8e/merged
shm 7.9M 4 7.9M 1% /var/lib/docker/containers/2289f61a0ea5776539bc7f0cb5d615ab9bb9ea6ebfc1aaeb55937a4690835c75/shm
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/1ff1ccba-4a2b-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/8c68d5a9bafaad34513cda8307f1891700e23d5ff00ae616d1b23a89a8ad966b/merged
shm 7.9M 4 7.9M 1% /var/lib/docker/containers/edd2bb34d731a7975dd37ba8b1579fc91368fb4b50ddabb6bc464c68a089ede7/shm
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/b53e6505-4a3a-11e8-8389-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/b77ba2cfb16a383b816eb79e91de7b8686275060078316fe678a8cc549ca9f41/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/e5e2cb318cf4063fb383d7acb22cd3443bfe9f5c60d92ae549f575dba24efee2/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/b3161b581184486a5e8b2e547cd603d7077fef7d5584cdedab13bd3bc2a24602/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/8f929839b62c2e46056917c51b05a0d93dabd6149c197bd1f5c911db4792584e/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/78c0c35fe5833d18b6d6d16ae81bd146910755bed831eb09adea3185796f8ff0/shm
overlay 118M 301K 118M 1% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs
overlay 118M 301K 118M 1% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs
tmpfs 7.9M 1 7.9M 1% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs/tmp
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/3bb5c6dbe8f7f46bb52b51fad949724c61417b51eeb12f68454067c296a481e1/merged
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/3d0587bd-4f6b-11e8-be69-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/3d1addd0-4f6b-11e8-be69-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-kjx5v
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/85ee038bfba77b3a046044b3b94fd2f789f0cec93e6e5414bc5913d09f03e3ff/merged
shm 7.9M 4 7.9M 1% /var/lib/docker/containers/d7736b64f084a4e61622247eef9b082df215cad901080d093f086a4656fd61e2/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/10fd5075134265a5367aabceb8aa1ffb46b45844d32092e6a56c102c41affd1b/merged
shm 7.9M 4 7.9M 1% /var/lib/docker/containers/2cf3a0373feb079701ba0c9cb003b2639e5447a6195954487a4d42fd7ab20c58/shm
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/0e55d061-4f73-11e8-be69-ce7be4be3aff/volumes/kubernetes.io~secret/kube-dns-token-66chm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/43032198af411e129e44889ceaa08824a01042fc1e1f11b62bbdc4ecdad5b6ae/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/fb0b8c81c0f6d3096f6f22dbc5a377e77b2e0dd28248f02d59d39b058d2f377c/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/43c4898ce4dea7e0a4670623da005ee215edbc9274d84a9ced1ccb8b581fa675/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/2ca7d653e8d622e1f58e55d3d27aa4e0adcef21d1152faa0a6108fc7eaf1e9b6/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/52af42a85e8c6e2d2a5e836bf684641c902f482d6fe81f5f2e9c797180701884/merged
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/b1d66af4-4f79-11e8-be69-ce7be4be3aff/volumes/kubernetes.io~secret/kube-dns-token-66chm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/7a8f6d65ec9583e368a8971dd4a512357bde7b0be7ff4f65fb11ab88e18ce527/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/ab5334cb4b295dbd161dc50aee32c5f301b6fa8329516b2e3b4ad0d8c94a14e7/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/962e030855e7493d5d8293900833ab1bcce18a7e90cc0cbc32ddbb99a134fcb0/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/f3633bc895a5433a6bc8599373a003a142bcc5a4a5d291d50e1381f66fe173f6/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/209ba4f77e2794e9373c46890596bbfef140232661f8144577ea2ab8db4f52e2/merged
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/50cb323a-5028-11e8-be69-ce7be4be3aff/volumes/kubernetes.io~secret/kube-dns-token-66chm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/b82c938c6f5387e7d0020a1707abcaf2b190649a992afb934af21534b0c0cdd5/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/71dd46e2f91605ca2854c2f812fa0117618126e86b3899ed41ee0a223cdd36f4/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/30aa11ade19c0d3edb016f932656217ada3d624827d938109afcee00927cf50e/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/41ca0c54012a9fccd140d4ffe7f8106e77a582f838af4d1aa11db5d191231f2e/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/378675550d7d737f2fce2a4cc8bcdb5dc41488f360babd0e282abd32e488af5d/merged
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/edebc31d-51cf-11e8-8636-ce7be4be3aff/volumes/kubernetes.io~secret/default-token-74f28
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/75eb35b761239195b7db7af509788ff4452f48357e2bb297db7d3badb066b129/merged
shm 7.9M 2 7.9M 1% /var/lib/docker/containers/c38bbf473810b5cf2a6dcf8b21d188b5cd9cf97119d1aa91903eee04aa773d3e/shm
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/f21f8584a02b60099827c609b1a1022e120ac98cca2c2969c81c62ad1802356e/merged
overlay 118M 301K 118M 1% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs/var/lib/docker/overlay2/de41fbe99da802b1167973c93fff008ea5ae661d6a01cf596b861241b5a93ec8/merged
overlay 118M 301K 118M 1% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs/var/lib/docker/overlay2/bf67f1130ccb20579b5f7c50a1d9f0d3459dae26fe2db8ad5923963de3ad75fb/merged
overlay 118M 301K 118M 1% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs/var/lib/docker/overlay2/bf67f1130ccb20579b5f7c50a1d9f0d3459dae26fe2db8ad5923963de3ad75fb/merged/root-disk/var/lib/docker/overlay2/bf67f1130ccb20579b5f7c50a1d9f0d3459dae26fe2db8ad5923963de3ad75fb/merged
overlay 118M 301K 118M 1% /var/lib/rkt/pods/run/738a7ece-95ec-4580-a55a-ef8e38955e7a/stage1/rootfs/opt/stage2/hyperkube/rootfs/var/lib/docker/overlay2/3f86e17b428a9bf2b4be02980a461e0ab0dd27476553c84ea5b3279d551b3f4f/merged
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/e1272c79-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/xxx-token-d445r
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/e1480a2a-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/xxx-token-d445r
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/9ddb2c64ab25cd2f8688b8fb9dd76287f78f45533df407a8e666af4b5f61a62f/merged
shm 7.9M 1 7.9M 1% /var/lib/docker/containers/e50f6e37a253e66699852c0e72ae4ae6eb7d741c67f1997388afe15de3e3b6b7/shm
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/e1d1fc17-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/xxx-token-d445r
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/e233b19b-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/xxx-token-d445r
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/e317c225-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/xxx-token-d445r
tmpfs 7.9M 9 7.9M 1% /var/lib/kubelet/pods/e28ab020-5445-11e8-8d53-ce7be4be3aff/volumes/kubernetes.io~secret/xxx-token-d445r
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/8981515939c853d28776631a9b4f251cc5086c3e9b5da775b606969cb189090e/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/74490bfdb71df7137b594d981dc2555602a4b9a5782c6218a74614993d0ac2da/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/9bc17139cac05e81a26c184a15b51722189dbcb10bdb03c618eb0bd6a50407b5/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/9183a101d103a2ef358185fde5a7ba4503b4c5a4e23fb791d2c015ad8cf63c19/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/cb885310d7f9bbe8090307135aee1e659b3ceee274061f49ed4f82b32d2db319/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/f35478fdb7882e70f5dca3c873c3d016d53047840004f83d91866b013a83383b/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/590b971b3bda3c8e9b3176e066a7200c77e560e9cab9c857cd8ba90d8f64d5fa/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/328c15e3613095d96f132b9c421ca9245c7e14f68740386ef1920ffd636d0d00/merged
overlay 118M 301K 118M 1% /var/lib/docker/overlay2/e76285f89bb14091f1cdc2aa4a511f1bb81fffd2f6b37a5580d10977623b04b5/merged
Hey, mind providing a bit more information and clarification?
In the issue title, you have "mounting shm tmpfs: no space left on device", but in the body you have "error creating overlay mount to /.../merged: no space left on device". How do those two errors relate? Once you get one, do you also start getting the other, or does a machine get into a state where they both start occurring?
I assume your docker daemon is in a default configuration, but if not, the output of docker info would be helpful.
Can you reproduce this reliably, or does it just sometimes start happening, and if you can reproduce it, how?
It might be helpful to check if the --all flag on df shows up anything interesting (though I wouldn't expect it to).
I am currently running into the same issue on multiple nodes running 1855.4.0 (stable).
Example of one of the failing pods:
kubectl -n monitoring describe po/prometheus-k8s-0
Name: prometheus-k8s-0
Namespace: monitoring
Priority: 0
PriorityClassName: <none>
Node: node-5.figo.systems/10.6.255.20
Start Time: Tue, 25 Sep 2018 13:09:51 +0200
Labels: app=prometheus
controller-revision-hash=prometheus-k8s-54ffbcbcf
prometheus=k8s
statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations: kubernetes.io/psp=restricted
Status: Running
IP: 10.6.68.17
Controlled By: StatefulSet/prometheus-k8s
Containers:
prometheus:
Container ID: docker://8d9f04d715569be99f97ce33369c3eaa72f698e81ee2274e5835b309600bfca6
Image: quay.io/prometheus/prometheus:v2.3.1
Image ID: docker-pullable://prom/prometheus@sha256:0283ae2509e1ccc71830edf382713cc3906aa55ca9418c50911854223829bf0b
Port: 9090/TCP
Host Port: 0/TCP
Args:
--web.console.templates=/etc/prometheus/consoles
--web.console.libraries=/etc/prometheus/console_libraries
--config.file=/etc/prometheus/config_out/prometheus.env.yaml
--storage.tsdb.path=/prometheus
--storage.tsdb.retention=30d
--web.enable-lifecycle
--storage.tsdb.no-lockfile
--web.route-prefix=/
State: Running
Started: Tue, 25 Sep 2018 13:10:15 +0200
Ready: True
Restart Count: 0
Liveness: http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
Readiness: http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120
Environment: <none>
Mounts:
/etc/prometheus/config_out from config-out (ro)
/etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
/prometheus from prometheus-persistent-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from prometheus-k8s-token-th9b4 (ro)
prometheus-config-reloader:
Container ID: docker://d8a700164e0bcd95dbc30e47ea0d3e3c415b4bf023b747a44f6475cfc39ce561
Image: quay.io/coreos/prometheus-config-reloader:v0.23.0
Image ID: docker-pullable://quay.io/coreos/prometheus-config-reloader@sha256:c7229ef9fb172ad15eb096d652f37badc49acea7080328a02a052a1ee343f998
Port: <none>
Host Port: <none>
Command:
/bin/prometheus-config-reloader
Args:
--log-format=logfmt
--reload-url=http://localhost:9090/-/reload
--config-file=/etc/prometheus/config/prometheus.yaml
--config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
State: Running
Started: Tue, 25 Sep 2018 13:10:22 +0200
Ready: True
Restart Count: 0
Limits:
cpu: 10m
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
POD_NAME: prometheus-k8s-0 (v1:metadata.name)
Mounts:
/etc/prometheus/config from config (rw)
/etc/prometheus/config_out from config-out (rw)
/var/run/secrets/kubernetes.io/serviceaccount from prometheus-k8s-token-th9b4 (ro)
rules-configmap-reloader:
Container ID: docker://8c48612b51a367b61b85d5fad1df0faa3e0fa05398f1f5ed64aba917ce6c6377
Image: quay.io/coreos/configmap-reload:v0.0.1
Image ID: docker-pullable://quay.io/coreos/configmap-reload@sha256:e2fd60ff0ae4500a75b80ebaa30e0e7deba9ad107833e8ca53f0047c42c5a057
Port: <none>
Host Port: <none>
Args:
--webhook-url=http://localhost:9090/-/reload
--volume-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
State: Waiting
Reason: CreateContainerError
Last State: Terminated
Reason: ContainerCannotRun
Message: error creating overlay mount to /var/lib/docker/overlay2/e4bf72696e188ca75187716c4583c2f967443c185be3ccf7957ec517244f4363/merged: no space left on device
Exit Code: 128
Started: Sat, 29 Sep 2018 11:12:14 +0200
Finished: Sat, 29 Sep 2018 11:12:14 +0200
Ready: False
Restart Count: 2
Limits:
cpu: 5m
memory: 10Mi
Requests:
cpu: 5m
memory: 10Mi
Environment: <none>
Mounts:
/etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from prometheus-k8s-token-th9b4 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
prometheus-persistent-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: prometheus-persistent-storage-prometheus-k8s-0
ReadOnly: false
config:
Type: Secret (a volume populated by a Secret)
SecretName: prometheus-k8s
Optional: false
config-out:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
prometheus-k8s-rulefiles-0:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: prometheus-k8s-rulefiles-0
Optional: false
prometheus-k8s-token-th9b4:
Type: Secret (a volume populated by a Secret)
SecretName: prometheus-k8s-token-th9b4
Optional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 13m (x7620 over 1d) kubelet, node-5.figo.systems (combined from similar events): Error: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/243f7d1eb9d0271f1368cbcc81bf82833d81b4d441d57e23e012accb8bd6978a-init/merged: no space left on device
Normal Pulled 3m (x7657 over 1d) kubelet, node-5.figo.systems Container image "quay.io/coreos/configmap-reload:v0.0.1" already present on machine
This is not unique to kubernetes, but to starting docker containers locally as well, e.g.:
Oct 01 07:03:37 node-6.figo.systems docker[54405]: /run/torcx/bin/docker: Error response from daemon: mounting shm tmpfs: no space left on device.
And here docker info:
Containers: 115
Running: 56
Paused: 0
Stopped: 59
Images: 37
Server Version: 18.06.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: v0.13.2 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
seccomp
Profile: default
selinux
Kernel Version: 4.14.67-coreos
Operating System: Container Linux by CoreOS 1855.4.0 (Rhyolite)
OSType: linux
Architecture: x86_64
CPUs: 64
Total Memory: 125.9GiB
Name: node-6.figo.systems
ID: MBHE:DHJX:Q7I2:SMBS:OBGU:6UWQ:GA3S:AJPY:DTY7:JJKY:Q3MG:A2ZT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
This issue might be related: https://github.com/moby/moby/issues/29638. On CoreOS the error is different though.
Some potentially helpful info:
core@node-6 ~ $ cat /proc/self/mountinfo | wc -l
25185
core@node-6 ~ $ mount | wc -l
25185
core@node-6 ~ $ docker ps -a | wc -l
117
core@node-6 ~ $ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 11 91 1
cpu 2 145 1
cpuacct 2 145 1
blkio 10 145 1
memory 5 1389 1
devices 4 145 1
freezer 9 91 1
net_cls 3 91 1
perf_event 7 91 1
net_prio 3 91 1
hugetlb 8 91 1
pids 6 156 1
core@node-6 ~ $ find /sys/fs/cgroup/memory/docker -type d | wc -l
3
core@node-6 ~ $ find /sys/fs/cgroup/memory -type d ! -path '/sys/fs/cgroup/memory/docker*' | wc -l
152
The root cause is that the resource is busy according to journalctl -eu docker:
...
Oct 01 08:03:12 node-6.figo.systems env[1837]: time="2018-10-01T08:03:12.903193735Z" level=error msg="Error removing mounted layer 5f18712935197fb61c86479ab40c15e7af7b3d1e652d2fe25a841d318d61cb11: remove /var/lib/docker/overlay2/6ee046bcb95387f41a58d576498f46fe8541830929cd09148e464d14786666c4/merged: device or resource busy"
Oct 01 08:03:12 node-6.figo.systems env[1837]: time="2018-10-01T08:03:12.903334746Z" level=error msg="Handler for DELETE /v1.31/containers/5f18712935197fb61c86479ab40c15e7af7b3d1e652d2fe25a841d318d61cb11 returned error: container 5f18712935197fb61c86479ab40c15e7af7b3d1e652d2fe25a841d318d61cb11: driver \"overlay2\" failed to remove root filesystem: remove /var/lib/docker/overlay2/6ee046bcb95387f41a58d576498f46fe8541830929cd09148e464d14786666c4/merged: device or resource busy"
Oct 01 08:03:25 node-6.figo.systems env[1837]: time="2018-10-01T08:03:25.779904862Z" level=error msg="Error removing mounted layer 0c4754d56f5aa974b2b3b4b0036ec1d10ebcd9573f63ee6eb7eb6b1d82e24021: remove /var/lib/docker/overlay2/b61f86d32867829c8a458ff5a1f56b09748661aae7571dd75dbdb504e92bb1c7/merged: device or resource busy"
Oct 01 08:03:25 node-6.figo.systems env[1837]: time="2018-10-01T08:03:25.780025983Z" level=error msg="Handler for DELETE /v1.31/containers/0c4754d56f5aa974b2b3b4b0036ec1d10ebcd9573f63ee6eb7eb6b1d82e24021 returned error: container 0c4754d56f5aa974b2b3b4b0036ec1d10ebcd9573f63ee6eb7eb6b1d82e24021: driver \"overlay2\" failed to remove root filesystem: remove /var/lib/docker/overlay2/b61f86d32867829c8a458ff5a1f56b09748661aae7571dd75dbdb504e92bb1c7/merged: device or resource busy"
Oct 01 08:03:38 node-6.figo.systems env[1837]: time="2018-10-01T08:03:38.593573706Z" level=error msg="Error removing mounted layer 644b6b8cd83eca92f6179de18bd1874feab626c0b347d167a74b5d3053d4f521: remove /var/lib/docker/overlay2/70e5657e93dbdf85b57f2f262797fd9387b6cc5b09ae4e333310aa1f7c1d4be1/merged: device or resource busy"
Oct 01 08:03:38 node-6.figo.systems env[1837]: time="2018-10-01T08:03:38.593700366Z" level=error msg="Handler for DELETE /v1.31/containers/644b6b8cd83eca92f6179de18bd1874feab626c0b347d167a74b5d3053d4f521 returned error: container 644b6b8cd83eca92f6179de18bd1874feab626c0b347d167a74b5d3053d4f521: driver \"overlay2\" failed to remove root filesystem: remove /var/lib/docker/overlay2/70e5657e93dbdf85b57f2f262797fd9387b6cc5b09ae4e333310aa1f7c1d4be1/merged: device or resource busy"
Oct 01 08:03:51 node-6.figo.systems env[1837]: time="2018-10-01T08:03:51.420397227Z" level=error msg="Error removing mounted layer 245b9fc2b3035d47a876934e9e2f01b147dc8bb14fd760a94fd7350faab00d2e: remove /var/lib/docker/overlay2/ef8040a41b4441fbbd2fa9557ff7a04a5fa784aeb9b150d0cfc97b9d7e95600d/merged: device or resource busy"
Oct 01 08:03:51 node-6.figo.systems env[1837]: time="2018-10-01T08:03:51.420562652Z" level=error msg="Handler for DELETE /v1.31/containers/245b9fc2b3035d47a876934e9e2f01b147dc8bb14fd760a94fd7350faab00d2e returned error: container 245b9fc2b3035d47a876934e9e2f01b147dc8bb14fd760a94fd7350faab00d2e: driver \"overlay2\" failed to remove root filesystem: remove /var/lib/docker/overlay2/ef8040a41b4441fbbd2fa9557ff7a04a5fa784aeb9b150d0cfc97b9d7e95600d/merged: device or resource busy"
Oct 01 08:04:04 node-6.figo.systems env[1837]: time="2018-10-01T08:04:04.206438606Z" level=error msg="Error removing mounted layer 64e8f782b15e5fe8d7010ac0da9b17ab73c7a352d2c0c1f3252124541c3a7543: remove /var/lib/docker/overlay2/c81399a149123655c1d477adb32ed8d2d0d1fb365b99664bc1603eed8b39042a/merged: device or resource busy"
Oct 01 08:04:04 node-6.figo.systems env[1837]: time="2018-10-01T08:04:04.206566990Z" level=error msg="Handler for DELETE /v1.31/containers/64e8f782b15e5fe8d7010ac0da9b17ab73c7a352d2c0c1f3252124541c3a7543 returned error: container 64e8f782b15e5fe8d7010ac0da9b17ab73c7a352d2c0c1f3252124541c3a7543: driver \"overlay2\" failed to remove root filesystem: remove /var/lib/docker/overlay2/c81399a149123655c1d477adb32ed8d2d0d1fb365b99664bc1603eed8b39042a/merged: device or resource busy"
Also interesting is that we never ran into this issue on 1800.7.0. Therefore this might be a regression caused by https://github.com/coreos/bugs/issues/2497? We are on a fairly young POC cluster therefore we might have not noticed before.
We found the issue it was fluent-bit still holding a handle to the logs (this applies to fluentd as well).
@trevex I'm not sure if this is 100% related since this thread seems to be a little older, but I'm running into some errors that I think closely resemble this thread. When I run df -h I also see thousands of mount points. I'm running mesosphere DCOS on CoreOS on Google Cloud. And I'm seeing this error on all the container-bearing agents:
journalctl -eu docker:
May 08 19:57:28 staging-aus-privateagent2 env[1929]: time="2019-05-08T19:57:28.732640790Z" level=error msg="error unmounting /var/lib/docker/overlay2/be4bdd0fe9a8e117d183b3ca42754feee5daabf78d25e70db48a88260e275d18-init/merged: invalid argument" storage-driver=overlay2
May 08 19:57:30 staging-aus-privateagent2 env[1929]: time="2019-05-08T19:57:30.083451553Z" level=error msg="Handler for POST /v1.38/containers/create returned error: error creating overlay mount to /var/lib/docker/overlay2/be4bdd0fe9a8e117d183b3ca42754feee5daabf78d25e70db48a88260e275d18-init/merged: no space left on device"
$ cat /proc/self/mountinfo | wc -l
49990
$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=2023.5.0
VERSION_ID=2023.5.0
BUILD_ID=2019-03-09-0138
PRETTY_NAME="Container Linux by CoreOS 2023.5.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
