docker
[BUG] Compose v2 is recreating ALL containers where there is a change to one of the "depends_on" in the hierarchy (bring back compose v1 behaviour)
Description
Separating out an issue described on https://github.com/docker/compose/issues/9600
Basically, this comment (https://github.com/docker/compose/issues/9600#issuecomment-1403124191) encompasses this issue. Sometimes containers are completely recreated when we do not expect them to, and when using "depends_on", and a dependant container changes in some way, all of the children are restarted. In compose v1.X, this is NOT the case, and ONLY the changed container is recreated and all others are untouched.
Either this is a bug, or a feature request to bring back compose v1 behaviour (at least with a flag).
Steps To Reproduce
Refer https://github.com/docker/compose/issues/9600#issuecomment-1403124191
Compose Version
Docker Compose version v2.16.0
Docker Environment
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
compose: Docker Compose (Docker Inc., v2.16.0)
scan: Docker Scan (Docker Inc., v0.21.0)
Server:
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 6
Server Version: 20.10.21
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 78f51771157abb6c9ed224c22013cdf09962315d
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-1160.80.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.06GiB
Name: XXX
ID: GR4X:Q35F:5E4H:TGFY:O34K:JEYN:WYBW:3QY5:4G7N:TXP6:4GLE:QOG2
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Anything else?
Refer to https://github.com/docker/compose/issues/9600 for alot of history.
With https://github.com/docker/compose/pull/10284 you can now explicitly opt-in/out for dependent container to be restarted when a service is recreated.
Can you please share how to configure this in compose file? Or link to the docs :) I couldn't see a reference in the MR.
see https://github.com/compose-spec/compose-spec/blob/master/spec.md#long-syntax-1
Hi @ndeloof , I finally got some time to test this today and it didn't work.
These are the steps I followed:
# compose ver:
Docker Compose version v2.17.0-rc.1
docker compose down
docker system prune --all
# clean slate for testing
# update docker-compose.yml as per example below
docker compose up -d
# pulls all 4 images and creates all 4 containers as expected
# Run "docker compose up -d" multiple times, no containers recreated needlessly - good!
docker compose up -d
docker compose up -d
docker compose up -d
# Then, I changed the image tag of service-4 in docker-compose.yml
docker compose up -d
# ALL the dependent containers got recreated :(
[+] Running 4/4
✔ Container dro-dns-fe-pdns-publisher-1 Running 0.0s
✔ Container dro-dns-fe-syslog-ng-1 Started 21.8s
✔ Container dro-dns-fe-pdns-recursor-1 Started 21.2s
✔ Container dro-dns-fe-dnsdist-1 Started 10.8s
# Run "docker compose up -d" multiple times, no containers recreated needlessly - good!
docker compose up -d
docker compose up -d
docker compose up -d
docker compose up -d
# Again, modified the image tag of service-4, ALL the dependent containers got recreated :(
# Run "docker compose up -d" multiple times, no containers recreated needlessly - good!
docker compose up -d
docker compose up -d
docker compose up -d
docker compose up -d
Sample docker-compose.yml file:
version: '3.7'
services:
service-1:
image: "XXXXXX.dkr.ecr.ap-southeast-2.amazonaws.com/image1:1.7.0-pdns-1026"
restart: always
env_file:
- .env_X
network_mode: host
depends_on:
service-2:
condition: service_started
restart: false
service-3:
condition: service_started
restart: false
service-2:
image: "XXXXXX.dkr.ecr.ap-southeast-2.amazonaws.com/image2:4.5.10-pdns-1023"
restart: always
env_file:
- .env_X
network_mode: host
volumes:
- /mnt/data/service-4/dev:/service-4
- /mnt/data/service-2:/var/service-2
depends_on:
service-4:
condition: service_started
restart: false
service-3:
condition: service_started
restart: false
service-3:
image: "XXXXXX.dkr.ecr.ap-southeast-2.amazonaws.com/image3:1.1.0-15-pdns-1024"
restart: always
env_file:
- .env_X
network_mode: host
volumes:
- /mnt/data/a:/a
service-4:
image: "XXXXXX.dkr.ecr.ap-southeast-2.amazonaws.com/image4:3.24-pdns-1022"
restart: always
env_file:
- .env_X
network_mode: host
volumes:
- /mnt/data/a/dev:/a
- /mnt/data/b:/b
- /mnt/data/c:/c
Have I made a mistake in the config, or is it just not working as expected?
Having same issue as decribed in https://github.com/docker/compose/issues/10308#issuecomment-1461332080 with Compose v2.24.6 (latest from Ubuntu 24.04 feed) Is this a new by design bahaviour or how to avoid this?
@andryushchenko see https://github.com/docker/compose/issues/10308#issuecomment-1447781552
@ndeloof Please can you provide the example of correct usage this option? I have the same situation as @captainfalcon23 - restart:false is set but conainers still being restarted
I have now spent some time looking into this issue, and as far as I can tell, the restart option of depends_on only applies to the docker compose restart command. At least, that's the only place that the implementation of #10284 touched. However, that does not come into effect when docker compose applies a new convergence on the entire dependency tree.
There's a pair of flags for docker compose up that are supposed to affect whether dependencies are recreated: --always-recreate-deps and --no-recreate; without the presence of either of these two, the recreation strategy should default to RecreateDiverged defined here, as selected by createOptions.dependenciesRecreateStrategy.
Unfortunately, the observed behavior seems to be as if --always-recreate-deps was on by default, but I'm not familiar enough with the internals of the convergence implementation to figure out why RecreateDiverged does not appear to apply.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
There's no indication that this has been addressed in any way. (I hope we don't have to play the stale / bump game every few months for years from here on out.)
This issue has been automatically marked as not stale anymore due to the recent activity.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
@koniiiik using this example compose file:
services:
service-1:
image: nginx
depends_on:
service-2:
condition: service_started
restart: false
service-2:
image: nginx
$ docker compose up -d
[+] Running 2/2
✔ Container truc-service-2-1 Started 0.3s
✔ Container truc-service-1-1 Started 0.3s
$ docker compose restart service-2
[+] Restarting 1/1
✔ Container truc-service-2-1 Started 0.2s
As you can see, with restart: false set, service-1 which depends on restarted service is not impacted.
I'm closing this issue as "fixed".
@ndeloof That is not what this issue is about, though. Yes, the restart: false option applies correctly when you restart a dependency. However, it has no effect if the dependency is recreated (for example, if you change the runtime configuration, or upgrade the image). As far as I can tell, there is currently no way to configure docker compose to avoid recreating all dependent services in this situation.
Here's an example that involves upgrading the tag in between docker compose up -d runs:
koniiiik@parahippus /tmp/docker-compose-example $ cat docker-compose.yml
services:
service-1:
image: nginx
depends_on:
service-2:
condition: service_started
restart: false
service-2:
image: nginx:1.27.3
koniiiik@parahippus /tmp/docker-compose-example $ docker compose up -d
[+] Running 3/3
✔ Network docker-compose-example_default Created 0.1s
✔ Container docker-compose-example-service-2-1 Started 0.3s
✔ Container docker-compose-example-service-1-1 Started 0.4s
koniiiik@parahippus /tmp/docker-compose-example $ cat docker-compose.yml
services:
service-1:
image: nginx
depends_on:
service-2:
condition: service_started
restart: false
service-2:
image: nginx:1.27.4
koniiiik@parahippus /tmp/docker-compose-example $ docker compose up -d
[+] Running 1/1
✔ service-2 Pulled 2.7s
[+] Running 2/2
✔ Container docker-compose-example-service-2-1 Started 0.5s
✔ Container docker-compose-example-service-1-1 Started 0.5s
koniiiik@parahippus /tmp/docker-compose-example $ docker compose ps -a
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
docker-compose-example-service-1-1 nginx "/docker-entrypoint.…" service-1 17 seconds ago Up 16 seconds 80/tcp
docker-compose-example-service-2-1 nginx:1.27.4 "/docker-entrypoint.…" service-2 17 seconds ago Up 16 seconds 80/tcp
I hope you can see that the issue has not been addressed, and that you reopen it.
indeed, thanks for clarification (it's hard to get a clear understanding on the many issues we have to address within my limited brain :P)
No worries, we've all been there. And at least now we have a truly minimal example to reproduce the issue.