[BUG] `--profile` breaking changes

[maciejpankanin]

Description

Imagine a stack like this:

services:
  db:
    image: postgres:13.3
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_USER: example
      POSTGRES_DB: example
    depends_on:
      hello-world:
        condition: service_completed_successfully

  hello-world:
    image: hello-world
    profiles:
      - hello

In previous versions of docker compose I could run docker compose up and it would start the db container without any problems. Recently I upgraded docker and was really surprised that it didn't work anymore. Here is the output of `docker compose up':

service hello-world is required by db, but is disabled. Can be enabled using profiles [hello]

This is a devastating change for many compose stacks, and wasn't even mentioned in the changelog :(

There is also another surprising behaviour when running docker compose logs db reports:

no such service: hello-world

Steps To Reproduce

1. Run docker compose up with the config above.

Compose Version

Any > 2.17.3(?)

Docker Environment

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 16
  Running: 0
  Paused: 0
  Stopped: 16
 Images: 266
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-32-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.22GiB
 Name: maciej-Latitude-5410
 ID: f4474bd0-abf8-4754-b75e-03c216f7b106
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[ndeloof]

ignoring the dependency was a bug, that we fixed. But as many did like you relied on this, we introduced required as a flag to mark a dependency as "nice to have", see https://github.com/docker/compose/pull/10792

[ndeloof]

Closing as a duplicate for https://github.com/compose-spec/compose-spec/issues/274

[maciejpankanin]

OK, thanks :)

But talking about this logs output, do you think it is OK?

[ndeloof]

indeed, I missed the "logs" issue as a sub-issue here :)

[GethDeeo]

While skimming through this issue and related PR, I came up with these two questions:

• When using up (without profiles), should the dependency check pass in case required dependencies are in profiles, but they're already up?
• When using down (without profiles), should the services without profiles be stopped and removed, regardless of any dependencies in profiles?

I'm trying to use a single instance of database as required dependency of multiple Compose projects. My idea was to merge the database's Compose file (via COMPOSE_FILES), and put the database service in a profile, so I don't accidentally destroy the database when playing with database-dependent Compose projects. Obviously it's not working this way, at least not in Compose 2.21.0.

[kmfarley11]

I'm gonna tag on to this thread for clarification reasons. In previous history, depends_on was ignored when running the build or pull commands.

i.e. I could have a service which depended on another for runtime reasons, but they were in different profiles to select which images to build at particular times etc.

However, with the latest docker compose, it appears that depends_on produces the "invalid compose project" command line error for build and pull now. Is that expected or unintended behavior?

For example:

services:
  # pretend like this service provides a cli to a web server, can be pulled from dockerhub
  cli:
    profiles: ["all", "cli", "pull"]
    image: busybox
    # since the cli is to be run against the web server, we mark as dependent
    # however, to build or pull the image, this shouldn't matter
    depends_on:
      web:
        condition: service_started

  # pretend like this webserver does stuff... and is built locally, not in "pull" profile
  web:
    profiles: ["all", "web"]
    image: nginx

If i just want to select images to be pulled, or built, I get errors where previous docker compose versions i did not (i.e. 2.3.3)

$ docker compose --profile pull pull
service "cli" depends on undefined service "web": invalid compose project
$ docker compose --profile cli build
service "cli" depends on undefined service "web": invalid compose project
$ docker compose version
Docker Compose version v2.27.0

If this is unintended and i need to open a separate issue just lmk please. (note: the workaround is to set required: false, but i thought build & pull weren't supposed to care about depends_on)