Example docker.service requires an externally managed containerd

[paralin]

Description

When upgrading to dockerd v23.0.0 the daemon stopped working due to timeouts contacting containerd via gRPC.

After unsetting the --containerd flag the old behavior is restored: the dockerd daemon starts containerd.

In docker.service:

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

v23.0.0 seems to have changed the systemd unit files with the expectation that containerd.service is installed and enabled alongside dockerd.service.

If the --containerd flag is set, dockerd will not start containerd automatically and expects it to be started independently.

I noticed that in the previous version 20.10.x this flag is not set in /usr/lib/systemd/system/docker.service:

ExecStart=/usr/bin/dockerd -H fd://

I couldn't find any mention of this change in the release notes: I would suggest adding something to the release notes documenting this change and the adjustments maintainers should make while upgrading to 23.0.0. (I'm maintaining the Buildroot packages for Docker).

Thanks!

Reproduce

1. Upgrade to docker v23.0.0 without adding containerd.service to the system
2. dockerd now fails to start with grpc timeout errors due to containerd not running

Expected behavior

Add a note to the release notes documenting this change

docker version

v23.0.0 (client and daemon).

Additional Info

define DOCKER_ENGINE_INSTALL_INIT_SYSTEMD
	$(INSTALL) -D -m 0644 $(@D)/contrib/init/systemd/docker.service \
		$(TARGET_DIR)/usr/lib/systemd/system/docker.service
	$(INSTALL) -D -m 0644 $(@D)/contrib/init/systemd/docker.socket \
		$(TARGET_DIR)/usr/lib/systemd/system/docker.socket
endef

[paralin]

I found the commit that introduced this change: 23c763ef1cb9fbe7a5b419cd6f6ab83eb5a69438

Even though the commit is from 2019, it was not included in 20.10.x and was included in 23.0.0 (checking the commit history).

CC @thaJeztah @mihao

Related change: 009639e0f46d270309821998836fb664046b3e07

[neersighted]

This is expected behavior; basically if you want a managed containerd, the daemon will manage the path based on the data-root. If you want an external containerd (the way most distros will, centrally managed by systemd) you pas the path via --containerd.

[thaJeztah]

I don't think this changed in the 23.0 release; the packaged versions of docker-ce already used this configuration since docker 18.09;

• https://github.com/docker/docker-ce-packaging/pull/298

The systemd unit in this repository were no longer used at that time (but were left in the repository in case someone would find them useful). https://github.com/moby/moby/commit/009639e0f46d270309821998836fb664046b3e07 was part of this PR:

• https://github.com/moby/moby/pull/42373

Which brought back the systemd service into this repository, but effectively no change in the packages (other than the packaging scripts now consuming the systemd unit(s) from this repository instead of from the packaging repository).

Note that without the --containerd option, the daemon would already prefer a running containerd service (if it finds a containerd socket in the default location), but this could sometimes cause a race condition if the containerd service started, but the socket not yet created, which could lead to two instances of containerd being started. The --containerd flag was added to prevent this race condition.

[paralin]

@thaJeztah The systemd unit file changed with this release, and we were/are using it in Buildroot. The issue is mainly just pointing out that this change ought to be mentioned in the release notes for maintainers of distros that are using the systemd unit inside this repo.