Container needlessly recreated and then won't start on docker-compose up

[Evesy]

Description

Following a change introduced in 2.3.4, most likely https://github.com/docker/compose/pull/9261, we're hitting a scenario where a container is recreated when it needn't be, and then will fail to start following that

Steps to reproduce the issue:

1. Create a docker-compose.yml similar to the below:

version: '3.7'
services:
  db:
    image: postgres:11
    hostname: postgres
    container_name: engage-db
    ports:
      - "5437:5432"
    environment:
      - POSTGRES_DB=somedb
      - POSTGRES_USER=someuser
      - POSTGRES_PASSWORD=somepassword

  wait-for-db:
    image: postgres:11
    command: [ sh, -c, "until pg_isready --username=someuser --host=db --port=5432; do sleep 5; done" ]
    links:
      - db

2. Ensure postgres:11 is not already in the docker image cache
3. Run docker-compose up -d db to start the db in the background, this will also pull the postgres:11 image.
4. Run docker-compose up wait-for-db to start the wait in the foreground

Describe the results you received: When the last step is run to start the wait-for-db container, it will result in the db container being recreated, despite the image etc. not having since been rebuilt or changed in any way:

$ docker-compose up wait-for-db
[+] Running 2/2
 ⠿ Container engage-db             Recreated                                                                                                     0.3s
 ⠿ Container engage-wait-for-db-1  Created                                                                                                       0.1s
Attaching to engage-wait-for-db-1

Following this, the recreated db container will also not be running, it will be sat in a Created state:

125fee896799   postgres:11   "docker-entrypoint.s…"   About a minute ago   Created                                engage-db

$docker inspect 125fee896799

``` [ { "Id": "125fee896799d4367c6ccae891065e120ded3ad6ccc8c525a45253abd5c88c27", "Created": "2022-04-06T10:05:51.95171076Z", "Path": "docker-entrypoint.sh", "Args": [ "postgres" ], "State": { "Status": "created", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 0, "Error": "", "StartedAt": "0001-01-01T00:00:00Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:04a660188f48f78bc42b2a03597123f9b0504f2db039daf8f2d98f93b04c2711", "ResolvConfPath": "", "HostnamePath": "", "HostsPath": "", "LogPath": "", "Name": "/engage-db", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "engage_default", "PortBindings": { "5432/tcp": [ { "HostIp": "", "HostPort": "5437" } ] }, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "CgroupnsMode": "private", "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IpcMode": "private", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "Mounts": [ { "Type": "volume", "Source": "26b0d26dc817cc36f1ad1290dfc27f1bc9984a9c5796a948eb3d6c9f27b2abe6", "Target": "/var/lib/postgresql/data" } ], "MaskedPaths": [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "ReadonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/10d96b1e87acb2e1617cc0c714fda6dfc727806d697033bd961637a3e372758d-init/diff:/var/lib/docker/overlay2/0d6e1a46381ea25e8cf96a9106f039946dd3e4427627e6cb1083005c718675e9/diff:/var/lib/docker/overlay2/4477f40fb0f241080e56a5450f965526ef4d7db71d62d81027cb1910b2e00f9d/diff:/var/lib/docker/overlay2/8bb3fbfc8a6bad7774fcac5117745a279a3d93a40cd8b1c4d577c34e2e9e5963/diff:/var/lib/docker/overlay2/112cd7cff09b5078cad860b6ae71dc9fb37e351656a229ac79f5ca68d0a2344b/diff:/var/lib/docker/overlay2/f6eb995351d9cbf225916217b20936bff5d4206931cddb1b9046cfeb22624ac7/diff:/var/lib/docker/overlay2/f5816c70b0e48705ce9827561cb91b99d67bb1671a341eb4101e329f780a13ba/diff:/var/lib/docker/overlay2/12453a6429f68bfe21e3b4f21b9febfb8117a896210dd2e29492001e5dc0bf1b/diff:/var/lib/docker/overlay2/42badba2cfaf487b90abdfc0417f3f814c0909f1b35242f0163a3535ccc23f8b/diff:/var/lib/docker/overlay2/ae6ff9f97ba34f0b9964492930f1e800e78ea65883e1b53d26e1136762895fde/diff:/var/lib/docker/overlay2/0f426746f21a5c0ddc48638a6b341190b522fce06b5b8f446760f33180556818/diff:/var/lib/docker/overlay2/fa6edca9880603046c4c5e5b65bbe59577cb7f6ec14264c81cf50c70600108bf/diff:/var/lib/docker/overlay2/9d45d8aedf1e0fcbe07c0c87822f8568c91d35f0dbb65e6cade336b000273725/diff:/var/lib/docker/overlay2/74b44bb155159c41af7b7ec4fad504f6753c0666d191bf2f4f234485e37d82b1/diff", "MergedDir": "/var/lib/docker/overlay2/10d96b1e87acb2e1617cc0c714fda6dfc727806d697033bd961637a3e372758d/merged", "UpperDir": "/var/lib/docker/overlay2/10d96b1e87acb2e1617cc0c714fda6dfc727806d697033bd961637a3e372758d/diff", "WorkDir": "/var/lib/docker/overlay2/10d96b1e87acb2e1617cc0c714fda6dfc727806d697033bd961637a3e372758d/work" }, "Name": "overlay2" }, "Mounts": [ { "Type": "volume", "Name": "26b0d26dc817cc36f1ad1290dfc27f1bc9984a9c5796a948eb3d6c9f27b2abe6", "Source": "/var/lib/docker/volumes/26b0d26dc817cc36f1ad1290dfc27f1bc9984a9c5796a948eb3d6c9f27b2abe6/_data", "Destination": "/var/lib/postgresql/data", "Driver": "local", "Mode": "z", "RW": true, "Propagation": "" } ], "Config": { "Hostname": "postgres", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "ExposedPorts": { "5432/tcp": {} }, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "POSTGRES_PASSWORD=somepassword", "POSTGRES_DB=somedb", "POSTGRES_USER=someuser", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/11/bin", "GOSU_VERSION=1.14", "LANG=en_US.utf8", "PG_MAJOR=11", "PG_VERSION=11.15-1.pgdg90+1", "PGDATA=/var/lib/postgresql/data" ], "Cmd": [ "postgres" ], "Image": "postgres:11", "Volumes": { "/var/lib/postgresql/data": {} }, "WorkingDir": "", "Entrypoint": [ "docker-entrypoint.sh" ], "OnBuild": null, "Labels": { "com.docker.compose.config-hash": "2cfb5e6a3de7283e5cf2ef6ff09ada28fbdc8c9447ae459da4f7fa2c00c1ec6c", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:04a660188f48f78bc42b2a03597123f9b0504f2db039daf8f2d98f93b04c2711", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "engage", "com.docker.compose.project.config_files": "/Users/Michael.Eves/Workspace/Docker/engage/docker-compose.yml", "com.docker.compose.project.working_dir": "/Users/Michael.Eves/Workspace/Docker/engage", "com.docker.compose.service": "db", "com.docker.compose.version": "2.3.4" }, "StopSignal": "SIGINT" }, "NetworkSettings": { "Bridge": "", "SandboxID": "", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "engage_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "engage-db", "db", "125fee896799", "postgres" ], "NetworkID": "", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "", "DriverOpts": null } } } } ] ```

If you stop everything docker-compose down -v and try again now the image postgres:11 is already pulled, the container is not recreated and everything works as expected.

Describe the results you expected: The container should not be recreated as nothing has changed about the image. Adding --no-recreate on the second compose command also works around the issue, but I don't believe the behaviour is correct currently on two points:

• The container shouldn't be recreated
• If a container is recreated, it should start back up

Output of docker compose version: Happens from vesion 2.3.4 inc. current release 2.4.1.

Output of docker info: 20.10.11/20.10.8

[ndeloof]

root cause is: as service image is pulled during up, com.docker.compose.image label is not set

[benlangfeld]

@ndeloof Your fix from #9366 was included in v2.5.0, but I can still reproduce this issue (com.docker.compose.image is unset after initial up) with docker compose v2.6.0. This issue should be re-opened.

[benlangfeld]

Indeed this bug is still reproducible in v2.10.1 (using docker:22.06.0-beta.0-dind).

[benlangfeld]

@ndeloof @ulyssessouza Could we please re-open this? It was not fixed by #9366.

[bibendi]

I also faced this issue. I'm using Docker Compose version v2.7.0. When I start my app with around 15 containers, just the mysql container is randomly recreated.

docker inspect dreg.***.io/dhub/library/mysql:5.7.32
[
    {
        "Id": "sha256:cc8775c0fe94cd6cbfc494688d09b078cda9bced9d9324d5c28bddf032344fc3",
        "RepoTags": [
            "dreg.***.io/dhub/library/mysql:5.7.32"
        ],
        "RepoDigests": [
            "dreg.***.io/dhub/library/mysql@sha256:e08834258fcc0efd01df358222333919df53d4a0d9b2a54da05b204b822e3b7b"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2021-01-12T10:12:44.131914824Z",
        "Container": "56ee61b1be5c0c698c895c3990f655333ec10c3fc44471b1b97ec88f1b49e6f0",
        "ContainerConfig": {
            "Hostname": "56ee61b1be5c",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3306/tcp": {},
                "33060/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GOSU_VERSION=1.12",
                "MYSQL_MAJOR=5.7",
                "MYSQL_VERSION=5.7.32-1debian10"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "CMD [\"mysqld\"]"
            ],
            "Image": "sha256:18ad8b374d80c80c3473ea052bf5dbf440ae0808066c0be850270dde54020746",
            "Volumes": {
                "/var/lib/mysql": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "DockerVersion": "19.03.12",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3306/tcp": {},
                "33060/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GOSU_VERSION=1.12",
                "MYSQL_MAJOR=5.7",
                "MYSQL_VERSION=5.7.32-1debian10"
            ],
            "Cmd": [
                "mysqld"
            ],
            "Image": "sha256:18ad8b374d80c80c3473ea052bf5dbf440ae0808066c0be850270dde54020746",
            "Volumes": {
                "/var/lib/mysql": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 448500822,
        "VirtualSize": 448500822,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/1d325c335062c051ae7268095e73680256061c3a690be2cf643a834f3a3f8f78/diff:/var/lib/docker/overlay2/e2eb630a87fc15ec4016b2fc66e336cd44993e662074095c9beb7cd88ae6f7f3/diff:/var/lib/docker/overlay2/d1438d837b19cab5d300131531327916c693de57b285aa7d161ed30f4bc39602/diff:/var/lib/docker/overlay2/a2f6aeda01e6a18ed1c9de625b81754d155e453f6404f37dbd1373f23b45c34d/diff:/var/lib/docker/overlay2/9649c7eb9315d3a0c5c9f2ea41d4677683715924459a69359919185b4920b37d/diff:/var/lib/docker/overlay2/cb7837f1a6a77992faee02b65080728b73a73a639b96aa254ab1383b51d866f1/diff:/var/lib/docker/overlay2/667070aeec58e97fff2cb8b20f3ab39238d52ccff7f84ef619fede73f3064b73/diff:/var/lib/docker/overlay2/44f7fc51144d497cd2933f4e1eba1c4a935cd60ff69215b8c0ff31093ba8b620/diff:/var/lib/docker/overlay2/b6871b7d5e24b1d415327330c6a5ac00267d3d9c1595430513126b34cd71723d/diff:/var/lib/docker/overlay2/79c371a05c2de598582db73608d65446660219c67c76ec466ce4a39a17f9709d/diff",
                "MergedDir": "/var/lib/docker/overlay2/3b920c483e4b0f94d0293df4ed15662a3e596415c62fbfa761ceac68ac911de5/merged",
                "UpperDir": "/var/lib/docker/overlay2/3b920c483e4b0f94d0293df4ed15662a3e596415c62fbfa761ceac68ac911de5/diff",
                "WorkDir": "/var/lib/docker/overlay2/3b920c483e4b0f94d0293df4ed15662a3e596415c62fbfa761ceac68ac911de5/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:cb42413394c4059335228c137fe884ff3ab8946a014014309676c25e3ac86864",
                "sha256:ef4a33cee7a02ffa3c1dd5d2618dcb95e37b899fb7e90d0bb2df32a37396e174",
                "sha256:74c86dffd46f095d4961730b731a33b19c1abd19f0a7ec1b81ddf09e6d3dfe85",
                "sha256:6d23902c2a54f36f80946d7225044b2681bac18570af3a58970e325b40ac0d60",
                "sha256:c484a3b6d84133f3618780f11216facc49f743cefc86e04409f28784a3a0a733",
                "sha256:0394a41efa739604258181808606eb47798a83383ff52241547583939ffb297c",
                "sha256:98d98806c8acefda16226c37045e3733a751c3cb165fde1182cbc324cde06f78",
                "sha256:8bdb6ee41f57372d9290e64d3de05a54f0f95cdd168dab725e39e96d9387cff3",
                "sha256:569a2d6e5302878870ada8b70e77a218ebad0659ad7d0d068187c1d320b15cea",
                "sha256:902dedce4cb4de96b4037a07e851f3f8b40c6c0ebc7172923cd58aa229a95862",
                "sha256:df1ee06a30c4b4463fc9f6ab182f6622f1b83b0e31f73468ce598fedc8b12564"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

[xyauheni-zededa]

i have the same issue on dc v2.6.1

[g-lashuk]

Docker Compose version v2.12.2 the issue is still represented

[captainfalcon23]

Seeing this issue on v2.14.0 too - any word on a fix? We have a use cae where we need to gradually add some services to the compose file and start them only after confirming the previous service is happy and healthy.

[ndeloof]

Tried to reproduce on latest codebase:

$ docker-compose --ansi=never up -d db
db Pulling 
45f14b91a40f Pulling fs layer 
...
0b9bf888f4ce Pull complete 
db Pulled 
Container engage-db  Creating
Container engage-db  Created
Container engage-db  Starting
Container engage-db  Started

$ docker-compose --ansi=never up wait-for-db
Container engage-db  Running
Container truc-wait-for-db-1  Creating
Container truc-wait-for-db-1  Created
Attaching to truc-wait-for-db-1
truc-wait-for-db-1  | db:5432 - accepting connections
truc-wait-for-db-1 exited with code 0

@captainfalcon23 you could use depends_on with a service_healthy condition for this use-case

[ndeloof]

Let's close this one as reproduction scenario doesn't help anymore and follow up on this with https://github.com/docker/compose/issues/9600