Respect depends_on conditions from docker-compose

[fgallese]

For reference, this issue was raised in #2210.

When using a docker compose file with one or more depends_on including conditions service_healthy, tilt does not check those conditions.

depends_on is a condition added in v2, and in v2.1 a condition service_healthy was added to it.

Example configuration:

services:
  myServiceA:
    extends:
      file: docker-compose.base.yml
      service: myServiceA_base
    depends_on:
      mariadb:
        condition: **service_healthy**
      mongodb:
        condition: **service_healthy**
      myServiceB:
        condition: service_started
    command: ["npm", "run", "dev"]

  mariadb:
    image: mariadb:10.1.30
    healthcheck:
      test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost"]
      timeout: 20s
      retries: 10

  mongodb:
    image: mongo:4.0
    healthcheck:
      test: echo 'db.runCommand("ping").ok' | mongo mongodb:27017/test --quiet
      interval: 10s
      timeout: 10s
      retries: 5

  myServiceB:
    extends:
      file: docker-compose.base.yml
      service: myServiceB_base
    depends_on:
      mariadb:
        condition: **service_healthy**
      mongodb:
        condition: **service_healthy**
    command: ["npm", "run", "dev"]

With this configuration, mariadb and mongodb should be started first, then myServiceB and at last myServiceA.

Tilt is not respecting this, as it is not checking the service_healthy conditions.

In docker compose v3 format, depends_on condition was removed altogether, but many companies are still using it, specially the ones that don't make use of Docker Swarm, so I think this should be supported by Tilt.

[ShadowLNC]

Just to add - the Compose Spec (which supersedes previous v2 and v3 formats) not only supports this, but requires it for compliance with the spec. We have some database migrations which rely on this feature, and they outright fail (and are not restarted as they're only meant to run once) because Tilt isn't waiting for the database to be healthy 😞

[Aryess]

I encountered this today as well, and got a minimal sample to reproduce the issue.

With the following docker-compose.yml:

---
version: "3"

services:
  slow:
    image: debian:stable-slim
    command: bash -c "echo 'Hi from slow container' && sleep 5 && echo 'Ok slow is ready now'"

  fast:
    image: debian:stable-slim
    command: "echo 'Hi from fast, this should be last'"
    depends_on:
      slow:
        condition: service_completed_successfully

And a Tiltfile containing just docker_compose('docker-compose.yml'), when doing docker compose up we get as expected:

tilt-depends-on|⇒ docker compose up
[+] Running 3/2
 ✔ Network tilt-depends-on_default   Created                                                                                         0.1s 
 ✔ Container tilt-depends-on-slow-1  Created                                                                                         0.0s 
 ✔ Container tilt-depends-on-fast-1  Created                                                                                         0.0s 
Attaching to tilt-depends-on-fast-1, tilt-depends-on-slow-1
tilt-depends-on-slow-1  | Hi from slow container
tilt-depends-on-slow-1  | Ok slow is ready now
tilt-depends-on-slow-1 exited with code 0
tilt-depends-on-fast-1  | Hi from fast, this should be last
tilt-depends-on-fast-1 exited with code 0

But when doing tilt up, we get:

Loading Tiltfile at: /tmp/tilt-depends-on/Tiltfile
Successfully loaded Tiltfile (73.417794ms)
         slow │ 
         slow │ Initial Build
         slow │ STEP 1/1 — Deploying
         slow │  Container tilt-depends-on-slow-1  Created
         slow │  Container tilt-depends-on-slow-1  Starting
         slow │  Container tilt-depends-on-slow-1  Started
         fast │ 
         fast │ Initial Build
         fast │ STEP 1/1 — Deploying
         fast │  Container tilt-depends-on-fast-1  Recreate
         slow │ Hi from slow container
         slow │ 
         slow │      Step 1 - 0.58s (Deploying)
         slow │      DONE IN: 0.58s 
         slow │ 
         fast │  Container tilt-depends-on-fast-1  Recreated
         fast │  Container tilt-depends-on-fast-1  Starting
         fast │  Container tilt-depends-on-fast-1  Started
         fast │ Hi from fast, this should be last
         fast │ 
         fast │      Step 1 - 0.88s (Deploying)
         fast │      DONE IN: 0.88s 
         fast │ 
         slow │ Ok slow is ready now

and we can see that the depends_on was not respected. Clearly, the current docker version (tested on docker 24.0.7 with compose 2.21.0) DOES support this. Looking at the documentation, it seems like "The depends_on option is ignored when deploying a stack in swarm mode". Is that what Tilt is doing? Would it be as simple as calling docker compose differently? I'm not sure, but that's all I could find on my end

[emersonfoni]

Just posting to try to keep this issue alive. We use Docker Compose and Tilt and have healthchecks, but resources fail to start because their dependencies aren't healthy yet, as "Tilt doesn’t currently observe docker-compose health checks". Engineers then have to repeatedly hit the reload button until the failed resources turns green.

Any chance this issue gets some love @nicks ?

[kinland]

I took a stab at this and opened a PR - it's probably not the solution that the maintainers would want, but as far as I can tell (well, at the level of energy for testing I have at 3am anyway), it solves this problem without side effects.