[BUG] flag --remove-orphans not working on Linux/Ubuntu

[oalagtash]

Description

I have two compose.yml files. One is the primary, and the second is just extending the services by adding a new profile. Important to say here, is that both compose files have the same name. I set that for both with COMPOSE_PROJECT_NAME=MyComposeName. I run compose up -d on the primary one. Then I run compose --profile integration-tests up -d on the second one. I navigate one directory back to the primary one and run docker compose ps. I see then see all services, including the ones activated by profile from the second compose.yml.

Now problem is: When I run docker compose down --remove-orphans in the primary directory, I expected the flag --remove-orphans to detect the services running in the second compose file and remove them.

Note that this strategy already works on MacOS. But not on Ubuntu.

Steps To Reproduce

1. on Ubuntu 22.04.3 LTS
2. two directories with compse.yml and .env file
3. in both .env files COMPOSE_PROJECT_NAME=MyProjectName
4. docker compose up -d in both directories
5. docker compose ps should show all services in both compose.ym files
6. docker compose down --remove-orphans does not remove services from other compose.yml file, even though they have the same compose project name.
7. this works on MacOS but not on Ubuntu

Compose Version

docker compose version:
Docker Compose version v2.24.5

docker-compose version:
Command 'docker-compose' not found, but can be installed with:
snap install docker          # version 24.0.5, or
apt  install docker-compose  # version 1.29.2-1
See 'snap info docker' for additional versions.

Docker Environment

Client: Docker Engine - Community
 Version:    25.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 44
  Running: 14
  Paused: 0
  Stopped: 30
 Images: 181
 Server Version: 25.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-92-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 94.26GiB
 Name: jenkins-prod
 ID: YAAZ:DOGG:CGMR:ERDY:F2RI:NAFN:6GHJ:2UKV:BM7C:QVGO:F7I7:ZUQQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[ndeloof]

docker compose ps list containers based on project name, not local compose.yaml files, so this is expected it also see the container started by another command/compose file

docker compose down --remove-orphans should also discover those containers, and I tried to reproduce your scenario without success: When using --remove-orphans I get additional containers removed. Could you please setup a minimal reproduction example ?

[oalagtash]

Yes, correct. With docker compose ps I was also able to see other containers started by another compose file. My problem is only with the docker compose down --remove-orphans command.

On Mac it works, but not on Linux/Ubuntu. Have you tested it there?

[ndeloof]

I only have a Mac computer to test, but I can't see any reason we would get a distinct behavior on Linux, especially as compose ps successfully list those "orphaned" containers (both commands use the same engine API).

[ndeloof]

I also tested running compose commands from within a docker image to be closer to your environment, still can't reproduce

[oalagtash]

This is exactly the problem. On my MacOS it works just fine. The exact same repository I checkout on the CI Server, which is Ubuntu, and it does not work. Maybe try using a virtual machine or spine up a server on the cloud?

[ndeloof]

Also tested inside a multipass ubuntu VM, still can't reproduce

[oalagtash]

Ok let me get back to you with minimal reproduction setup

[Gornoka]

I have a reproduction of at least a very similar issue @ndeloof . If you think this is worth another issue let me know.

https://github.com/Gornoka/compose-rm-test

[Gornoka]

if you rollback to the following versions you see the old / correct behaviour, where it remove these containers, which were orphaned due to changed profiles.

VERSION_STRING=5:24.0.6-1~ubuntu.20.04~focal
COMPOSE_VERSION=2.21.0-1~ubuntu.20.04~focal
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin=$COMPOSE_VERSION

[ndeloof]

ok, I eventually understood your confusion here: As you run docker compose down with an compose.yaml file, the command knows your model and knows about defined services (including those with disabled profiles). Such services are not considered orphaned as they exist in the model (but disabled). "Orphaned" containers are those which have no definition in the compose model, in most case after compose file was updated, + container rans as one-off with docker compose run.

[Gornoka]

Why did this behaviour change?

Also i did not see any note to this breaking change in the documentation

If it is now a defined feature of compose to treat these containers as "not orphans" that is fine by me, but it still messes up the restart of those containers since the network reference is outdated, which I think should then be addressed.

[ndeloof]

see https://github.com/docker/compose/pull/11204 listed in release note https://github.com/docker/compose/releases/tag/v2.23.2

it still messes up the restart of those containers since the network reference is outdated, which I think should then be addressed.

Can you please clarify which scenario you have in mind ?