Stopping containers doesn't work on Ubuntu 23.10

[jfbauer432]

Describe the bug

Several weeks ago I noticed errors from watchtower when updating other containers. At first I thought the container was taking too long to exit, so I added a --stop-timeout=60s to the watchtower command, but that didn't help. Looking more info the logs, I see that it gets the error right way after trying to stop the container. This happens when trying to update different containers. I can manually restart then just fine.

Not certain, but this issue might have started after upgrading from Ubuntu 23.04 to Ubuntu 23.10. Or possibly after the most recent watchtower update

REPOSITORY                     TAG           IMAGE ID       CREATED         SIZE
containrrr/watchtower          latest        e7dd50d07b86   6 weeks ago     14.7MB

Steps to reproduce

1. docker run --rm --name watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --debug --run-once unifi (see full log below)
2. gets error

time="2023-12-27T13:38:08Z" level=info msg="Stopping /unifi (ff08acd24e6b) with SIGTERM"
time="2023-12-27T13:38:08Z" level=error msg="Error response from daemon: Cannot kill container: ff08acd24e6b453fcbb32c8842e4f6fe73d5e1f75091bcc6896d74602ea47a82: Cannot kill container ff08acd24e6b453fcbb32c8842e4f6fe73d5e1f75091bcc6896d74602ea47a82: unknown error after kill: runc did not terminate successfully: exit status 1: unable to signal init: permission denied\n: unknown"
time="2023-12-27T13:38:08Z" level=info msg="Session done" Failed=1 Scanned=1 Updated=0 notify=no

Expected behavior

Expected that watchtower can successfully kill (and restart) containers.

Screenshots

No response

Environment

• Platform Ubuntu 23.10
• Architecture x86_64
• Docker Version 24.0.5, build 24.0.5-0ubuntu1

Your logs

$ docker run --rm --name watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --debug --run-once unifi
time="2023-12-27T13:38:06Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2023-12-27T13:38:07Z" level=debug msg="Making sure everything is sane before starting"
time="2023-12-27T13:38:07Z" level=info msg="Watchtower 1.7.1"
time="2023-12-27T13:38:07Z" level=info msg="Using no notifications"
time="2023-12-27T13:38:07Z" level=info msg="Only checking containers which name matches \"unifi\""
time="2023-12-27T13:38:07Z" level=info msg="Running a one time update."
time="2023-12-27T13:38:07Z" level=debug msg="Checking containers for updated images"
time="2023-12-27T13:38:07Z" level=debug msg="Retrieving running containers"
time="2023-12-27T13:38:07Z" level=debug msg="Trying to load authentication credentials." container=/unifi image="jacobalberty/unifi:latest"
time="2023-12-27T13:38:07Z" level=debug msg="No credentials for index.docker.io found" config_file=/config.json
time="2023-12-27T13:38:07Z" level=debug msg="Got image name: jacobalberty/unifi:latest"
time="2023-12-27T13:38:07Z" level=debug msg="Checking if pull is needed" container=/unifi image="jacobalberty/unifi:latest"
time="2023-12-27T13:38:07Z" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/"
time="2023-12-27T13:38:07Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2023-12-27T13:38:07Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2023-12-27T13:38:07Z" level=debug msg="Setting scope for auth token" image=docker.io/jacobalberty/unifi scope="repository:jacobalberty/unifi:pull"
time="2023-12-27T13:38:07Z" level=debug msg="No credentials found."
time="2023-12-27T13:38:07Z" level=debug msg="Parsing image ref" host=index.docker.io image=jacobalberty/unifi normalized=docker.io/jacobalberty/unifi tag=latest
time="2023-12-27T13:38:07Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/jacobalberty/unifi/manifests/latest"
time="2023-12-27T13:38:08Z" level=debug msg="Found a remote digest to compare with" remote="sha256:23d3114328fcb21fd7a0ae37a38d7b65b4e21adc653b58aeadc1e3aecc4b30d6"
time="2023-12-27T13:38:08Z" level=debug msg=Comparing local="sha256:250536053f6902b324e0cf181e8a4601d8bc2e94287f42bcc7999732abadd45c" remote="sha256:23d3114328fcb21fd7a0ae37a38d7b65b4e21adc653b58aeadc1e3aecc4b30d6"
time="2023-12-27T13:38:08Z" level=debug msg="Digests did not match, doing a pull."
time="2023-12-27T13:38:08Z" level=debug msg="Pulling image" container=/unifi image="jacobalberty/unifi:latest"
time="2023-12-27T13:38:08Z" level=info msg="Found new jacobalberty/unifi:latest image (ec36c92163a6)"
time="2023-12-27T13:38:08Z" level=info msg="Stopping /unifi (ff08acd24e6b) with SIGTERM"
time="2023-12-27T13:38:08Z" level=error msg="Error response from daemon: Cannot kill container: ff08acd24e6b453fcbb32c8842e4f6fe73d5e1f75091bcc6896d74602ea47a82: Cannot kill container ff08acd24e6b453fcbb32c8842e4f6fe73d5e1f75091bcc6896d74602ea47a82: unknown error after kill: runc did not terminate successfully: exit status 1: unable to signal init: permission denied\n: unknown"
time="2023-12-27T13:38:08Z" level=info msg="Session done" Failed=1 Scanned=1 Updated=0 notify=no
time="2023-12-27T13:38:08Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

Additional context

No response

[github-actions[bot]]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[danst0]

Same here, came after my upgrade to ubuntu 23.10.

[piksel]

Yeah, seems to be related to apparmor permissions. Perhaps @simskij has some insights?

[jfbauer432]

Yeah, seems to be related to apparmor permissions.

Here is a kernel log that happened when watchtower got the error

kernel: [510121.642686] audit: type=1400 audit(1703840408.407:47786): apparmor="DENIED" operation="signal" class="signal" profile="docker-default" pid=1330911 comm="runc" requested_mask="receive" denied_mask="receive" signal=term peer="/usr/sbin/runc"

[piksel]

This bug contains some more info (not on Ubuntu 23.10, so I cannot test it unfortunately): https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/2039294

I guess we need to allow watchtower/docker to send signals using an apparmor rule somehow. I have very little experience with apparmor though :/

[jfbauer432]

Looks like this change fixes the issue

diff -u -r /etc/apparmor.d.bak/docker /etc/apparmor.d/docker
--- /etc/apparmor.d.bak/docker  2024-01-05 13:51:22.718451513 -0500
+++ /etc/apparmor.d/docker      2024-01-05 13:59:18.445871143 -0500
@@ -37,4 +37,5 @@
   # suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
   ptrace (trace,read) peer=docker-default,
 
+  signal receive set=(term kill) peer="/usr/sbin/runc",
 }

and then tell apparmor to reload the modified profile by running

sudo apparmor_parser -r /etc/apparmor.d/docker

[danst0]

I don't have a /etc/apparmor.d/docker profile, according to aa-status I have a docker-default profile, however cannot find that in /etc/apparmor.d/

ps. from my short research I think the correct way would be to create a new profile and pass that to Watchtower as a security option (https://docker-docs.uclv.cu/engine/security/apparmor/)

[lukaszzyla]

same issue here - ubuntu 23.10

[Quinten0508]

Docker automatically generates and loads a default profile for containers named docker-default. On Docker versions 1.13.0 and later, the Docker binary generates this profile in tmpfs and then loads it into the kernel. On Docker versions earlier than 1.13.0, this profile is generated in /etc/apparmor.d/docker instead.

Since the default AppArmor profile is generated dynamically you will not be able to edit it, or (afaik) even read its current contents. Could anyone on an older version share their /etc/apparmor.d/docker so we can use it as template to create a modified AppArmor profile for watchtower?

[lukaszzyla]

No need for that. Just reinstall docker and everything works...

Wysłano z Samsung w Plus Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Quinten @.> Sent: Sunday, January 21, 2024 3:18:53 PM To: containrrr/watchtower @.> Cc: lukaszzyla @.>; Comment @.> Subject: Re: [containrrr/watchtower] Stopping containers doesn't work on Ubuntu 23.10 (Issue #1891)

Docker automatically generates and loads a default profile for containers named docker-default. On Docker versions 1.13.0 and later, the Docker binary generates this profile in tmpfs and then loads it into the kernel. On Docker versions earlier than 1.13.0, this profile is generated in /etc/apparmor.d/docker instead.

Since the default AppArmor profile is generated dynamically you will not be able to edit it, or (afaik) even read its current contents. Could anyone on an older version share their /etc/apparmor.d/docker so we can use it as template to create a modified AppArmor profile for watchtower?

— Reply to this email directly, view it on GitHubhttps://github.com/containrrr/watchtower/issues/1891#issuecomment-1902643859, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI2CO7LPAOPYKVH5AFPILNLYPUPU3AVCNFSM6AAAAABBEMODP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBSGY2DGOBVHE. You are receiving this because you commented.Message ID: @.***>

[danst0]

Ok, a little bit more context would be great. what did you do? Apt reinstall docker.io did not work.

[lukaszzyla]

Google for it. Basically apt remove packages and apt install them again. Do not remove your volumes and images and everything you already have will still be there afterwards. Sorry i cannot provide you the exact link now but this should be enough sudo apt-get remove docker docker-engine docker.io And then reinstall. Or find a good guide on installing docker from scratch and do it backwards first.

Wysłano z Samsung w Plus Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: danst0 @.> Sent: Sunday, January 21, 2024 7:31:24 PM To: containrrr/watchtower @.> Cc: lukaszzyla @.>; Comment @.> Subject: Re: [containrrr/watchtower] Stopping containers doesn't work on Ubuntu 23.10 (Issue #1891)

Ok, a little bit more context would be great. what did you do? Apt reinstall docker.io did not work.

— Reply to this email directly, view it on GitHubhttps://github.com/containrrr/watchtower/issues/1891#issuecomment-1902721429, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI2CO7JR746KA2YHOWBTCVDYPVNHZAVCNFSM6AAAAABBEMODP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBSG4ZDCNBSHE. You are receiving this because you commented.Message ID: @.***>

[danst0]

Ok. My install of 23.10 is quite new. Why should that work?Am 21.01.2024 um 22:15 schrieb lukaszzyla @.***>: Google for it.

Basically apt remove packages and apt install them again.

Do not remove your volumes and images and everything you already have will still be there afterwards.

Sorry i cannot provide you the exact link now but this should be enough

sudo apt-get remove docker docker-engine docker.io

And then reinstall.

Or find a good guide on installing docker from scratch and do it backwards first.

Wysłano z Samsung w Plus

Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: danst0 @.***>

Sent: Sunday, January 21, 2024 7:31:24 PM

To: containrrr/watchtower @.***>

Cc: lukaszzyla @.>; Comment @.>

Subject: Re: [containrrr/watchtower] Stopping containers doesn't work on Ubuntu 23.10 (Issue #1891)

Ok, a little bit more context would be great. what did you do? Apt reinstall docker.io did not work.

—

Reply to this email directly, view it on GitHubhttps://github.com/containrrr/watchtower/issues/1891#issuecomment-1902721429, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI2CO7JR746KA2YHOWBTCVDYPVNHZAVCNFSM6AAAAABBEMODP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBSG4ZDCNBSHE.

You are receiving this because you commented.Message ID: @.***>

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[lukaszzyla]

There is some problem with docker default profile. Happened to me after update. Nevertheless - uninstalling and reinstalling should create new default.profile - at lear that was my thinking after seeing the problem. And it was solved as currently watchtower does the job correctly.

Wysłano z Samsung w Plus Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: danst0 @.> Sent: Monday, January 22, 2024 6:03:48 AM To: containrrr/watchtower @.> Cc: lukaszzyla @.>; Comment @.> Subject: Re: [containrrr/watchtower] Stopping containers doesn't work on Ubuntu 23.10 (Issue #1891)

Ok. My install of 23.10 is quite new. Why should that work?Am 21.01.2024 um 22:15 schrieb lukaszzyla @.***>: Google for it.

Basically apt remove packages and apt install them again.

Do not remove your volumes and images and everything you already have will still be there afterwards.

Sorry i cannot provide you the exact link now but this should be enough

sudo apt-get remove docker docker-engine docker.io

And then reinstall.

Or find a good guide on installing docker from scratch and do it backwards first.

Wysłano z Samsung w Plus

Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: danst0 @.***>

Sent: Sunday, January 21, 2024 7:31:24 PM

To: containrrr/watchtower @.***>

Cc: lukaszzyla @.>; Comment @.>

Subject: Re: [containrrr/watchtower] Stopping containers doesn't work on Ubuntu 23.10 (Issue #1891)

Ok, a little bit more context would be great. what did you do? Apt reinstall docker.io did not work.

—

Reply to this email directly, view it on GitHubhttps://github.com/containrrr/watchtower/issues/1891#issuecomment-1902721429, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI2CO7JR746KA2YHOWBTCVDYPVNHZAVCNFSM6AAAAABBEMODP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBSG4ZDCNBSHE.

You are receiving this because you commented.Message ID: @.***>

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHubhttps://github.com/containrrr/watchtower/issues/1891#issuecomment-1903262858, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI2CO7PCJ5QGRE62KBKC5BLYPXXLJAVCNFSM6AAAAABBEMODP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBTGI3DEOBVHA. You are receiving this because you commented.Message ID: @.***>

[danst0]

There is not package docker or docker-engine. reinstall did not work. Still looking for a solution.

[lukaszzyla]

Install Docker Engine on Ubuntu | Docker Docshttps://docs.docker.com/engine/install/ubuntu/

just follow the first commands to uninstall everything. do not forget sudo

for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done

---

From: danst0 @.> Sent: 23 January 2024 9:01 PM To: containrrr/watchtower @.> Cc: lukaszzyla @.>; Comment @.> Subject: Re: [containrrr/watchtower] Stopping containers doesn't work on Ubuntu 23.10 (Issue #1891)

There is not package docker or docker-engine. reinstall did not work. Still looking for a solution.

— Reply to this email directly, view it on GitHubhttps://github.com/containrrr/watchtower/issues/1891#issuecomment-1906829496, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI2CO7NH6C7YPEGMANQO423YQAJIVAVCNFSM6AAAAABBEMODP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBWHAZDSNBZGY. You are receiving this because you commented.Message ID: @.***>

[AnthonySchuijlenburg]

Doing the complete reinstall (but skipping the cleanup of old containers/images/volumes) worked for me.

[danst0]

Complete uninstall, reinstall didn't work here.

[lukaszzyla]

hi, sorry to hear that. This is strange, as we have the similar setup and similar problem. I am also on ubuntu 23.10 and it did the trick for me. are you sure you haven't tried other solutions that could have influenced your setup/config? maybe your problem is also connected to missing apparmor? Can you check sudo apparmor_status ? I can see that watchtower has docker-default profile defined in apparmor_status: 41 processes have profiles defined. 41 processes are in enforce mode. /watchtower (1480) docker-default I am not a specialist in linux but I remember I did install apparmor which I did not have before.

---

From: danst0 @.> Sent: 24 January 2024 8:54 PM To: containrrr/watchtower @.> Cc: lukaszzyla @.>; Comment @.> Subject: Re: [containrrr/watchtower] Stopping containers doesn't work on Ubuntu 23.10 (Issue #1891)

Complete uninstall, reinstall didn't work here.

— Reply to this email directly, view it on GitHubhttps://github.com/containrrr/watchtower/issues/1891#issuecomment-1908818474, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AI2CO7PJWVOPBBDUIIIL653YQFRHLAVCNFSM6AAAAABBEMODP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBYHAYTQNBXGQ. You are receiving this because you commented.Message ID: @.***>

[danst0]

root@cassius # apt list apparmor
Auflistung... Fertig
apparmor/mantic,now 4.0.0~alpha2-0ubuntu5 amd64  [installiert]

Seems to be.

root@cassius4 /h/# apparmor_status | grep docker
   docker-default
   /package/admin/s6-2.11.3.2/command/s6-svscan (6499) docker-default
   /package/admin/s6-2.11.3.2/command/s6-supervise (6586) docker-default
[...]

[lukaszzyla]

I can't be of much help here. for me apt list states:

lukasz@chw-homeserver:~$ sudo apt list apparmor
Listing... Done
apparmor/mantic,now 4.0.0~alpha2-0ubuntu5 amd64 [installed,automatic]
`
[installed,automatic] - I have no clue why and if it does make any difference...

[KingOperator]

Hi, it stopped working for me since upgrading to Ubuntu 23.10 too. Reinstall didn't work as well. Tried:

sudo apt remove docker*
sudo apt autoremove
sudo apt install docker.io docker-compose-v2

sudo apparmor_status | grep watchtower says: /watchtower (3014113) docker-default

apt list docker* apparmor* -i says:

apparmor/mantic,now 4.0.0~alpha2-0ubuntu5 amd64  [installiert]
docker-compose-v2/mantic,now 2.20.2+ds1-0ubuntu1 amd64  [installiert]
docker.io/mantic,now 24.0.5-0ubuntu1 amd64  [installiert]

The file /etc/apparmor.d/docker doesn't exist.

[LeGEC]

As a temporary patch on my system, I disabled the apparmor rules for /usr/sbin/runc

Following the documentation to disable one single apparmor profile (link on help.ubuntu.com: Disable one profile ) :

sudo ln -s /etc/apparmor.d/usr.sbin.runc /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.runc

docker can now send signals to its containers.

---

re-activating is documented in the next paragraph in the page above:

sudo rm /etc/apparmor.d/disable/usr.sbin.runc
sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.runc

docker stop will not be able to send a signal to its containers anymore.

(repost of my comment on the launchpad issue)

---

note: if anyone has a better fix (something that would only target the action of runc on docker containers for example ?) I'm interested

[kprinssu]

@LeGEC There has been some activity on the Launchpad Issue and the fixes have been merged in the upstream projects (Moby and containerd).

The fix that worked for me is to do the following:

1. Edit or create a new file at /etc/apparmor.d/docker-default with the following contents:

#include <tunables/global>


profile docker-default flags=(attach_disconnected,mediate_deleted) {

  #include <abstractions/base>


  network,
  capability,
  file,
  umount,
  # Host (privileged) processes may send signals to container processes.
  signal (receive) peer=unconfined,
  # dockerd may send signals to container processes (for "docker kill").
  signal (receive) peer=unconfined,
  # runc may send signals to container processes
  signal (receive) peer=runc,
  # Container processes may send signals amongst themselves.
  signal (send,receive) peer=docker-default,

  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  # deny write to files not in /proc/<number>/** or /proc/sys/**
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9/]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,

  deny mount,

  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/devices/virtual/powercap/** rwklx,
  deny /sys/kernel/security/** rwklx,

  # suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
  ptrace (trace,read,tracedby,readby) peer=docker-default,
}

2. Tell AppArmor to load the new profile via apparmor_parser -Kr /etc/apparmor.d/docker-default

[ocampeau]

I had a similar issue with a bare metal installation of Kubernetes. After upgrading to Ubuntu 23.10, my cluster was in a bad state and rebooting it with kubeadm failed with the same error:

runc did not terminate successfully: exit status 1: unable to signal init: permission denied

My solution was not only to re-install containerd, which is the CRI I use, but to also delete all container mounts under /var/lib/containerd/k8s/.

Re-installing containerd was not enough. I had to delete all previous containers too.

[Garfonso]

@kprinssu Thank you very much for posting the solution here. First, it did not work for me, and I was a bit frustrated because I've been searching for months. I managed to look into the apparmor message about the denial today and saw that for me the peer is not runc but, for some reason, /usr/sbin/runc which seems to make a difference. So I adjusted the line with peer=runc to peer=/usr/sbin/runc.

Yay, finally. :-)