support restarting docker?

[huguesdk]

would it be possible to adjust the configuration so that the services are still up (or properly restarted) when docker is restarted?

i use this playbook on a server that is used for other services, and the owner of the server often updates the packages when updates are available. when docker is updated, it is restarted, and because of this all matrix services are stopped. is it possible to ensure that they are properly running after that automatically?

what do i need to do for this to be the case?

[spantaleev]

The fact that all our systemd services have configuration like this, should make them auto-restart when Docker restarts:

Requires=docker.service
After=docker.service

---

I have, myself, also seen services not auto-restart after a Docker upgrade though..

I've just done some manual testing and none of these break it (that is, Matrix services restart as expected):

• systemctl restart docker
• systemctl restart containerd

However, the following sequence makes services stop and never start again: systemctl stop docker && systemctl start docker.

I suspect the package upgrade procedure involves a full service stop like this, an upgrade of the package files, and then a start.

I wonder if making our services dependent on containerd.service would help.. If the upgrade procedure for that one is the same, then it likely leads to the same issue.

[christianlupus]

@spantaleev This is wrong what you are saying as far as I understand. By After= you tell systemd to start the services after docker if docker.service is to be started. So it only sets the ordering. By Requires= you set a dependency on docker. That is, to start e.g. the synapse docker container, docker must be running. Thus, there is a dependency from synapse to docker.

All these are unidirectional in the sense that for e.g. synapse, docker is needed and started before it. Contrary, if docker is pulled down, systemd might most probably pull down the dependent services as well. However there is no requirement that the containers are restarted (by the means of systemd) after docker has been stopped and started.

On the other side there is the docker flag to restart unless-stopped or always. This should be sufficient in general even if docker is stopped and started again (even a reboot is ok). But if systemd pulls down the dependent service before docker can save the state, this will not help much....

[spantaleev]

We use --rm though, the manual for which says:

--rm flag can work together with -d, and auto-removal will be done on daemon side. Note that it's incompatible with any restart policy other than none

Not sure if that's only when used in combination with -d (which we don't use) or even without it.

---

Specifying a restart policy also seems like the wrong way to do things, given that we'd like to manage the service via systemd. Yes, systemd just manages a proxy to the actual container process and not the real process, but still, introducing one more manager process sounds odd.. Hmm..

[christianlupus]

I did not want to offend here. I have no clear solution. I just read your post and thought I saw an error in your argumentation that might give a clue to the issue.

[spantaleev]

Sure! Thanks for clearing things up and suggesting some potential solution!

Given that we have some way to reproduce the problem (systemctl stop docker && systemctl start docker), we should be able to test if it helps or not.

[Tuinslak]

Updating docker also breaks the auto-restart function.

[SuperSandro2000]

This sounds like a perfect match for live restore. https://docs.docker.com/config/containers/live-restore/

[jimeh]

I'm no systemd expert, but based on the README, is sounds like systemd-docker potentially solves all these restart issues: https://github.com/ibuildthecloud/systemd-docker

[spantaleev]

That project looks nice! It has the the following downsides though:

• we'd need to get it installed via the playbook
• it seems like it may not work well on CentOS 7
• it's unmaintained and the last change is from 7 years ago, so other things may break with newer systemd/Docker versions on at least one or more of the distros we support

[jimeh]

@spantaleev right, yeah those downsides are not ideal. I might try and find some time to do a bit of research into running containers as system services.

I want to transition my own playbooks and machines away from Ansible's docker_container module to systemd, hence this issue is interesting to me, and something I'd be curious to try and solve :)