for-linux icon indicating copy to clipboard operation
for-linux copied to clipboard
verified publisher icon docker

Core dump of containerd.service on 5:20.10.12~3-0~debian-bullseye_armhf.deb (RasPi ZeroW)

Open LorenAmelang opened this issue 3 years ago • 11 comments

  • [x] This is a bug report
  • [ ] This is a feature request
  • [x] I searched existing issues before opening this one

Expected behavior

Installation of containerd.io_1.4.12-1_armhf.deb 2021-12-11 23:01:38 14.7 MiB docker-ce-cli_20.10.12~3-0~debian-bullseye_armhf.deb 2021-12-13 14:39:56 34.0 MiB docker-ce-rootless-extras_20.10.12~3-0~debian-bullseye_armhf.deb 2021-12-13 14:39:58 6.3 MiB docker-ce_20.10.12~3-0~debian-bullseye_armhf.deb 2021-12-13 14:39:58 12.6 MiB on Raspberry Pi Zero W with fresh Debian Bullseye

Actual behavior

A dependency job for docker.service failed. See 'journalctl -xe' for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com

Jan 23 16:19:43 raspberrypi systemd[1]: Dependency failed for Docker Application Container Engine.
Jan 23 16:19:43 raspberrypi systemd[1]: docker.service: Job docker.service/start failed with result 'dependency'.
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for libc-bin (2.31-13+rpt2+rpi1+deb11u2) ...
Errors were encountered while processing:
 docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)
pi@raspberrypi:~ $

@raspberrypi:~ $ journalctl -xe
░░
░░ A stop job for unit containerd.service has finished.
░░
░░ The job identifier is 143343 and the job result is done.
Jan 23 16:22:41 raspberrypi systemd[1]: Starting containerd container runtime...
░░ Subject: A start job for unit containerd.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd.service has begun execution.
░░
░░ The job identifier is 143343.
Jan 23 16:22:41 raspberrypi systemd[1]: containerd.service: Main process exited, code=dumped, statu>
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit containerd.service has exited.
░░
░░ The process' exit code is 'dumped' and its exit status is 11.
Jan 23 16:22:41 raspberrypi systemd[1]: containerd.service: Failed with result 'core-dump'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit containerd.service has entered the 'failed' state with result 'core-dump'.
Jan 23 16:22:41 raspberrypi systemd[1]: Failed to start containerd container runtime.
░░ Subject: A start job for unit containerd.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd.service has finished with a failure.
░░
░░ The job identifier is 143343 and the job result is failed.
pi@raspberrypi:~ $

Steps to reproduce the behavior

[https://docs.docker.com/engine/install/debian/] (https://docs.docker.com/engine/install/debian/)

Exploration:

pi@raspberrypi:~ $ ls -al /lib/systemd/system/
total 1028
drwxr-xr-x 21 root root 20480 Jan 23 16:19  .
drwxr-xr-x 17 root root  4096 Dec 16 20:40  ..
...
-rw-r--r--  1 root root  1162 Apr  2  2021  apparmor.service
...
-rw-r--r--  1 root root  1263 Nov 17 10:48  containerd.service
-rw-r--r--  1 root root  1254 Aug  6 20:58  [email protected]
...
-rw-r--r--  1 root root  1695 Dec 13 03:44  docker.service
-rw-r--r--  1 root root   175 Dec 13 03:44  docker.socket
...
-rw-r--r--  1 root root   404 Feb  2  2021  sockets.target
drwxr-xr-x  2 root root  4096 Oct 30 04:13  sockets.target.wants


pi@raspberrypi:~ $ sudo find / -name "*containerd.service*"
/usr/lib/systemd/system/containerd.service
/var/lib/systemd/deb-systemd-helper-enabled/containerd.service.dsh-also
/var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants/containerd.service
/etc/systemd/system/multi-user.target.wants/containerd.service
pi@raspberrypi:~ $


pi@raspberrypi:~ $ systemctl status containerd.service
● containerd.service - containerd container runtime
     Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: core-dump) since Sun 2022-01-23 16:42:55 PST; 1s ago
       Docs: https://containerd.io
    Process: 6759 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 6760 ExecStart=/usr/bin/containerd (code=dumped, signal=SEGV)
   Main PID: 6760 (code=dumped, signal=SEGV)
        CPU: 366ms

Jan 23 16:42:55 raspberrypi systemd[1]: containerd.service: Failed with result 'core-dump'.
Jan 23 16:42:55 raspberrypi systemd[1]: Failed to start containerd container runtime.
pi@raspberrypi:~ $

Output of docker version:

pi@raspberrypi:~ $ docker version
Segmentation fault
pi@raspberrypi:~ $

Output of docker info:

pi@raspberrypi:~ $ docker info
Segmentation fault
pi@raspberrypi:~ $

Additional environment details (AWS, VirtualBox, physical, etc.) This is exactly the 2018 problem reported in: https://forums.docker.com/t/error-when-starting-docker-service/63041/4

gforghetti
Gary Forghetti
Docker Staff
Feb '19
That is a known issue if you installed Docker 18.09.1.
The workaround is to install 18.06.03 till a patch release becomes available for 18.09.

But I have docker-ce (5:20.10.12~3-0~debian-bullseye) ... And it isn't 2019 any more...

https://github.com/docker/for-linux/issues/490

thaJeztah commented on Oct 10, 2019
This should be resolved now with the container.io_1.2.10-2 package

But I have containerd.io is already the newest version (1.4.12-1)

LorenAmelang avatar Jan 24 '22 01:01 LorenAmelang

I gave up on Docker and tried to install my NodeJS project directly, but the Pi was constantly repeating the containerd core dump. I tried every command I could find to disable Docker and containerd, but nothing worked. I finally removed Docker and containerd:

Jan 25 16:57:18 raspberrypi systemd[1]: containerd.service: Failed with result 'core-dump'.
Jan 25 16:57:18 raspberrypi systemd[1]: Failed to start containerd container runtime.
Jan 25 16:57:23 raspberrypi systemd[1]: containerd.service: Scheduled restart job, restart counter is at 1669.
Jan 25 16:57:24 raspberrypi systemd[1]: Stopped containerd container runtime.
Jan 25 16:57:24 raspberrypi systemd[1]: Starting containerd container runtime...
Jan 25 16:57:24 raspberrypi systemd[1]: containerd.service: Main process exited, code=dumped, status=11/SEGV
Jan 25 16:57:24 raspberrypi systemd[1]: containerd.service: Failed with result 'core-dump'.
Jan 25 16:57:24 raspberrypi systemd[1]: Failed to start containerd container runtime.
Jan 25 16:57:29 raspberrypi systemd[1]: containerd.service: Scheduled restart job, restart counter is at 1670.
Jan 25 16:57:30 raspberrypi systemd[1]: Stopped containerd container runtime.
Jan 25 16:57:30 raspberrypi systemd[1]: Reloading.
Jan 25 16:57:33 raspberrypi systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.
Jan 25 16:57:35 raspberrypi systemd[1]: Reloading.
Jan 25 16:57:38 raspberrypi systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.
Jan 25 16:57:43 raspberrypi systemd[1]: Reloading.
Jan 25 16:57:46 raspberrypi systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.

Is there some command I didn't find that would have worked to stop the thousands of core dumps?

LorenAmelang avatar Jan 27 '22 06:01 LorenAmelang

same here just with

Apr 22 11:14:24 raspberrypi systemd[1]: containerd.service: Main process exited, code=dumped, status=4/ILL Apr 22 11:14:24 raspberrypi systemd[1]: containerd.service: Failed with result 'core-dump'. Apr 22 11:14:24 raspberrypi systemd[1]: Failed to start containerd container runtime.

This kernel-version is the problem

Chiller2019 avatar Apr 22 '22 09:04 Chiller2019

Same with newer versions of docker-ce and containerd:

schizophrenia@cloudberry:/var/log $ docker ps
Segmentation fault
schizophrenia@cloudberry:/var/log $ dpkg -l | grep container
ii  containerd.io                        1.5.11-1                         armhf        An open and reliable container runtime
iF  docker-ce                            5:20.10.14~3-0~debian-bullseye   armhf        Docker: the open-source application container engine
ii  docker-ce-cli                        5:20.10.14~3-0~debian-bullseye   armhf        Docker CLI: the open-source application container engine

Also on raspbian on zero 1.1:

schizophrenia@cloudberry:~ $ lsb_release -a
uname No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye
schizophrenia@cloudberry:~ $ uname -a
Linux cloudberry 5.15.32+ #1538 Thu Mar 31 19:37:58 BST 2022 armv6l GNU/Linux

epikhinm avatar May 01 '22 17:05 epikhinm

@Chiller2019

This kernel-version is the problem

Why do you have such guess?

epikhinm avatar May 01 '22 17:05 epikhinm

@Chiller2019

This kernel-version is the problem

Why do you have such guess?

Because I read in many forums that, when anybody change (downgrade) the kernel-version, its work. I dont know why but it works for me.

Chiller2019 avatar May 01 '22 19:05 Chiller2019

@Chiller2019

This kernel-version is the problem

Why do you have such guess?

Because I read in many forums that, when anybody change (downgrade) the kernel-version, its work. I dont know why but it works for me.

What the kernel version do you use? Are you use it on raspbian? How do you downgrade it?

epikhinm avatar May 01 '22 20:05 epikhinm

same issue with raspi zero

xmmwc avatar May 28 '22 10:05 xmmwc

I switched to podman, as workaround.

epikhinm avatar Jun 06 '22 09:06 epikhinm

I am having this issue as well.

mathmaniac43 avatar Nov 05 '22 01:11 mathmaniac43

Happening for me using DietPi on a Pi Zero W after installing the latest using the instructions from Docker's website.

journalctl -xe

Log Paste 
Dec 07 02:47:19 enviro systemd[1]: Starting containerd container runtime...
░░ Subject: A start job for unit containerd.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd.service has begun execution.
░░
░░ The job identifier is 43164.
Dec 07 02:47:19 enviro sudo[2019]:   dietpi : TTY=pts/0 ; PWD=/home/dietpi ; USER=root ; COMMAND=/usr/bin/su -Dec 07 02:47:19 enviro sudo[2019]: pam_unix(sudo:session): session opened for user root(uid=0) by dietpi(uid=1000)
Dec 07 02:47:19 enviro su[2020]: (to root) dietpi on pts/0
Dec 07 02:47:19 enviro su[2020]: pam_unix(su-l:session): session opened for user root(uid=0) by dietpi(uid=0)
Dec 07 02:47:20 enviro systemd[1]: containerd.service: Main process exited, code=dumped, status=11/SEGV
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit containerd.service has exited.
░░
░░ The process' exit code is 'dumped' and its exit status is 11.
Dec 07 02:47:20 enviro systemd[1]: containerd.service: Failed with result 'core-dump'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit containerd.service has entered the 'failed' state with result 'core-dump'.
Dec 07 02:47:20 enviro systemd[1]: Failed to start containerd container runtime.
░░ Subject: A start job for unit containerd.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd.service has finished with a failure.
░░
░░ The job identifier is 43164 and the job result is failed.

Packages

root@enviro:~# dpkg -l  | grep docker
iF  docker-ce                     5:20.10.21~3-0~debian-bullseye   armhf        Docker: the open-source application container engine
ii  docker-ce-cli                 5:20.10.21~3-0~debian-bullseye   armhf        Docker CLI: the open-source application container engine
ii  docker-compose-plugin         2.12.2~debian-bullseye           armhf        Docker Compose (V2) plugin for the Docker CLI.
root@enviro:~# dpkg -l | grep containerd
ii  containerd.io                 1.6.10-1                         armhf        An open and reliable container runtime

Kernel Version

root@enviro:~# uname -a
Linux enviro 5.15.76+ #1597 Fri Nov 4 12:11:43 GMT 2022 armv6l GNU/Linux

LSB release info

root@enviro:~# lsb_release  -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

dkowis avatar Dec 07 '22 02:12 dkowis

Hello,

For me this also happens on bullseye_armhf.deb RasPi ZeroW. A dependency job for docker.service failed.

uname -a Linux raspberry 6.1.21+ #1642 Mon Apr 3 17:19:14 BST 2023 armv6l GNU/Linux

Kralg avatar May 09 '24 12:05 Kralg