docker
[BUG] 'docker-compose up' fails before a container's dependency is healthy
Discussed in https://github.com/docker/compose/discussions/11064
Originally posted by sergiohcis October 4, 2023 Hi Comunity, My problem is that I have some services and they have dependencies between them, like this:
services:
service1:
image: sergio/service1:latest
healthcheck:
test: "curl --insecure -I https://service1/actuator/health | grep 200 || exit 1"
interval: 10s
timeout: 5s
retries: 5
service2:
image: sergio/service2:latest
healthcheck:
test: "curl --insecure -I https://service2/actuator/health | grep 200 || exit 1"
interval: 10s
timeout: 5s
retries: 5
depends_on:
service1:
condition: service_healthy
'service2' depends on 'service1' and thing is that service1 is taking som time to be healthy, so after all the retries of healthcheck, it shows "unhealthy", but some time later it shows: "healthy" Before I was using docker-compose v2.2.3, and it always watied until 'service1' was "healthy" then started "service2" without failing. Now, I updated docker-compose to V2.22.0, and I am getting this error: "dependency failed to start: container service1 is unhealthy", then it stops and never starts 'service2'. Is this the expected behaviour ? What can I do to force it to wait as it was doing before?
Thank you.
I've run into the same issue and I wanted to raise this as an issue since this makes configuration of certain stacks a nightmare (eg VPN container establishing a connection before a second container is allowed to start), and there has been no comment for months.
docker version
Client: Docker Engine - Community
Version: 25.0.3
API version: 1.44
Go version: go1.21.6
Git commit: 4debf41
Built: Tue Feb 6 21:13:11 2024
OS/Arch: linux/arm64
Context: default
Server: Docker Engine - Community
Engine:
Version: 25.0.3
API version: 1.44 (minimum version 1.24)
Go version: go1.21.6
Git commit: f417435
Built: Tue Feb 6 21:13:11 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.28
GitCommit: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker context show
default
docker info
Client: Docker Engine - Community
Version: 25.0.3
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.5
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 6
Running: 4
Paused: 0
Stopped: 2
Images: 7
Server Version: 25.0.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: active
NodeID: cn2zge78o9x228zlwin6sqeyt
Is Manager: false
Node Address: 192.168.1.5
Manager Addresses:
192.168.1.4:2377
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-1045-raspi
Operating System: Ubuntu 22.04.3 LTS
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 1.801GiB
Name: alyssa
ID: ab0574d4-a364-4ec4-ae57-61e3b69eb043
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
As you get dependency failed to start: container service1 is unhealthy reported, can you please check using docker ps the health status reported by docker engine ? If the latter is "unhealthy" then this just is the expected behavior. Maybe then you can tweak the healthcheck parameters (especially, using start_period and start_interval) to give your service enough time to get ready
@ndeloof why is this expected behavior? I would expect compose to wait indefinitely for the dependent service to become healthy, rather than exiting. The behavior isn't documented in the description of the healthcheck option.
healthcheck is a runtime feature, not a compose one. Compose is only used here to configure runtime. Health comes with a timeout (30 seconds by default), so compose will not wait indefinitely - which is also expected: healthcheck role is not to wait for container to be ready, whatever time it takes, but to let user know if container is up to provide the expected service.
I'm closing this issue as "works as expected"
The ability to create dependencies between contains is a compose feature, not a runtime feature. There's no good reason for compose to exit when a container is unhealthy. You can close this if you want, but it doesn't change the fact that the current behavior is surprising, pointless, and undocumented.
Rather than closing this issue, it should be resolved either through a documentation pull request so that the current behavior is explicitly described, or preferably via a code pull request that either changes the default behavior or introduces a new compose option (--wait-forever?) that would result in the more obvious behavior.