compose icon indicating copy to clipboard operation
compose copied to clipboard
verified publisher icon docker

Sporadic failures stopping a postgres container since 2.10.0 (error connecting to docker daemon)

Open nocive opened this issue 3 years ago • 2 comments

Description

When stopping a container via docker compose stop we started observing sporadic failures doing so when using docker compose versions >= 2.10.0

Steps to reproduce the issue:

  1. docker compose start --wait postgres
  2. <seed postgres with some data>
  3. run docker compose stop and receive an error
  4. run docker compose stop a second time and it succeeds.

Describe the results you received: Summarized the output for brevity reasons, see below for the full (but truncated) output.

12:03:24  Container workspace-postgres-1  Stopping
(...)
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Full output 
12:03:24  Container workspace-postgres-1  Stopping
(...)
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
(...)
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
(...)
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Error while Stopping
12:03:24  Container workspace-postgres-1  Stopped
(...)
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopping
12:03:24  Container workspace-postgres-1  Stopped
(...)
12:03:24  Container workspace-postgres-1  Stopped
12:03:24  Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Describe the results you expected: container stopped successfully like it happens with <= 2.9.x versions.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker compose version:

Docker Compose version v2.10.2

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.10.2)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 29
  Running: 1
  Paused: 0
  Stopped: 28
 Images: 112
 Server Version: 20.10.18
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.0-13-cloud-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 14.29GiB
 Name: xxx
 ID: 77HA:5L4O:ZGQI:FIAM:3BBY:FE33:NEZJ:O2ZY:VQBL:X4EX:SQUA:4UIN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: xxx
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

This is the definition for our postgres container:

postgres:
    image: postgres:12.7-alpine
    stdin_open: true
    tty: true
    command: -c max_connections=100
    environment:
      POSTGRES_HOST_AUTH_METHOD: trust
    expose:
      - 5432
    ports:
      - "${HOST_ADDR}:5432:5432"
    healthcheck:
      test:     "pg_isready -q -d postgres -U postgres"
      interval: 30s
      timeout:  5s
      retries:  5
    volumes:
      - /home/fulliautomatix/.workspace/volumes/postgres/data:/var/lib/postgresql/data:delegated
      - /home/fulliautomatix/.workspace/volumes/downloads:/shared/downloads:delegated
      - /home/fulliautomatix/.workspace/volumes/scripts:/shared/scripts:ro

We only observe this problem with the 2.10.x versions, with earlier versions this problem does not occur at all, so the culprit was introduced somewhere in 2.10.0.

We're using a stop timeout of 30s but tinkering with this value seems to make no difference.

So far we haven't found a way to consistently reproduce the issue, therefore I can't provide a minimal reproduction scenario neither.

nocive avatar Sep 12 '22 11:09 nocive

I've got a few questions to try and help narrow this down somewhat:

  1. What is the newest version of Compose that you don't see the issue with? v2.9.0?
  2. Have you only downgraded Compose and seen the problem go away? (I am asking because I also see you are on the latest engine release, v20.10.18, which is quite new, so I want to make sure we're isolating changes)
  3. Is it happening for all containers or just Postgres?

milas avatar Sep 12 '22 15:09 milas

hi @milas :wave:

  1. yes, 2.9.0 and lower do not exhibit this issue;
  2. yes, compose version seems to be the only source for the problem, problem existed with docker 20.10.17 too.
  3. just postgres, but we don't really stop any other containers during our CI pipelines (where we can observe the issue).

nocive avatar Sep 12 '22 15:09 nocive

I don't know what it was, but it seems to have been fixed somewhere in 2.11.1 or 2.11.2, we were not able to reproduce with any of these versions. Closing.

nocive avatar Oct 04 '22 13:10 nocive

I'm glad to hear it and thanks for following up. If it comes back up, feel free to re-open or create a new issue.

milas avatar Oct 05 '22 17:10 milas