Default network is not removed if startedServices is used

[menulis]

If startedServices is used as part of dockerCompse block to specify which services from docker-compose.yml need to be started, the default network created as part of composeUp is not removed as part of composeDown. Otherwise, if startedServices is not used, the default network is removed as expected.

docker-compose.yml:

version: "3.7"

services:

  redis:
    image: redis:4-alpine
    ports:
      - 6379
    volumes:
      - redisdata:/data
    environment:
      - ALLOW_EMPTY_PASSWORD=yes

  influxdb:
    image: influxdb:1.8-alpine
    volumes:
      - influxdata:/var/lib/influxdb
    ports:
      - 8086

  firestore-emulator:
    ports:
      - 9090
    build:
      context: ./emulators/firestore

  pubsub-emulator:
    ports:
      - 9091
    build:
      context: ./emulators/pubsub

volumes:
  influxdata:
  redisdata:

• Scenario without startedServices:

dockerCompose {
    isRequiredBy(tasks.test.get())
}

$ ../gradlew test
> Task :phone-backend:composeUp
redis uses an image, skipping
influxdb uses an image, skipping
Building firestore-emulator
Building pubsub-emulator
Creating network "269e031a17ca122b18dda2fa930b457e_phone-backend__default" with the default driver
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1 ... 
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... 
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... 
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1           ... 
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1           ... done
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... done
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1    ... done
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... done
Will use localhost as host of redis
Will use localhost as host of influxdb
Will use localhost as host of firestore-emulator
Will use localhost as host of pubsub-emulator
Probing TCP socket on localhost:55061 of 'redis_1'
TCP socket on localhost:55061 of 'redis_1' is ready
Probing TCP socket on localhost:55058 of 'influxdb_1'
TCP socket on localhost:55058 of 'influxdb_1' is ready
Probing TCP socket on localhost:55059 of 'firestore-emulator_1'
TCP socket on localhost:55059 of 'firestore-emulator_1' is ready
Probing TCP socket on localhost:55060 of 'pubsub-emulator_1'
TCP socket on localhost:55060 of 'pubsub-emulator_1' is ready
+----------------------+----------------+-----------------+
| Name                 | Container Port | Mapping         |
+----------------------+----------------+-----------------+
| redis_1              | 6379           | localhost:55061 |
+----------------------+----------------+-----------------+
| influxdb_1           | 8086           | localhost:55058 |
+----------------------+----------------+-----------------+
| firestore-emulator_1 | 9090           | localhost:55059 |
+----------------------+----------------+-----------------+
| pubsub-emulator_1    | 9091           | localhost:55060 |
+----------------------+----------------+-----------------+

> Task :phone-backend:composeDown
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1           ... 
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1    ... 
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... 
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... 
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1           ... done
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... done
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1    ... done
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... done
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1           ... 
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1    ... 
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... 
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... 
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... done
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1           ... done
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1    ... done
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... done
Removing network 269e031a17ca122b18dda2fa930b457e_phone-backend__default
Removing volume 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdata
Removing volume 269e031a17ca122b18dda2fa930b457e_phone-backend__redisdata

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 36s
14 actionable tasks: 2 executed, 12 up-to-date

$ docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
43539f28c077   bridge    bridge    local
bd8e2d8de947   host      host      local
d86aaf7e317d   none      null      local

• Scenario with startedServices:

dockerCompose {
    isRequiredBy(tasks.test.get())
    startedServices = listOf(
        "redis",
        "influxdb",
        "firestore-emulator",
        "pubsub-emulator"
    )
}

$ ../gradlew test           

> Task :phone-backend:composeUp
redis uses an image, skipping
influxdb uses an image, skipping
Building firestore-emulator
Building pubsub-emulator
Creating network "269e031a17ca122b18dda2fa930b457e_phone-backend__default" with the default driver
Creating volume "269e031a17ca122b18dda2fa930b457e_phone-backend__influxdata" with default driver
Creating volume "269e031a17ca122b18dda2fa930b457e_phone-backend__redisdata" with default driver
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1 ... 
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... 
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... 
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1    ... 
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1    ... done
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1           ... done
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... done
Creating 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... done
Will use localhost as host of redis
Will use localhost as host of influxdb
Will use localhost as host of firestore-emulator
Will use localhost as host of pubsub-emulator
Probing TCP socket on localhost:55063 of 'redis_1'
TCP socket on localhost:55063 of 'redis_1' is ready
Probing TCP socket on localhost:55065 of 'influxdb_1'
TCP socket on localhost:55065 of 'influxdb_1' is ready
Probing TCP socket on localhost:55064 of 'firestore-emulator_1'
TCP socket on localhost:55064 of 'firestore-emulator_1' is ready
Probing TCP socket on localhost:55062 of 'pubsub-emulator_1'
TCP socket on localhost:55062 of 'pubsub-emulator_1' is ready
+----------------------+----------------+-----------------+
| Name                 | Container Port | Mapping         |
+----------------------+----------------+-----------------+
| redis_1              | 6379           | localhost:55063 |
+----------------------+----------------+-----------------+
| influxdb_1           | 8086           | localhost:55065 |
+----------------------+----------------+-----------------+
| firestore-emulator_1 | 9090           | localhost:55064 |
+----------------------+----------------+-----------------+
| pubsub-emulator_1    | 9091           | localhost:55062 |
+----------------------+----------------+-----------------+

> Task :phone-backend:composeDown
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... 
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... 
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1    ... 
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1           ... 
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1           ... done
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... done
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1    ... done
Stopping 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... done
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... 
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... 
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1    ... 
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1           ... 
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1    ... done
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1              ... done
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1           ... done
Removing 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1 ... done
Going to remove 269e031a17ca122b18dda2fa930b457e_phone-backend__redis_1, 269e031a17ca122b18dda2fa930b457e_phone-backend__firestore-emulator_1, 269e031a17ca122b18dda2fa930b457e_phone-backend__pubsub-emulator_1, 269e031a17ca122b18dda2fa930b457e_phone-backend__influxdb_1

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 30s
14 actionable tasks: 2 executed, 12 up-to-date

$ docker network ls
NETWORK ID     NAME                                                      DRIVER    SCOPE
7f08646fd28e   269e031a17ca122b18dda2fa930b457e_phone-backend__default   bridge    local
43539f28c077   bridge                                                    bridge    local
bd8e2d8de947   host                                                      host      local
d86aaf7e317d   none                                                      null      local

gradle-docker-compose-plugin version used: 0.13.4

[augi]

Hello, thank you for the report!

Unfortunately, this is the expected behavior, just look at code here.

The root cause is that docker-compose down doesn't support selecting services to stop/remove, so we have to mimic the behavior using docker-compose rm but this doesn't support networks removal.

Originally, we invoked docker-compose down even if startedServices was specified but it leads to the removal of services that weren't started by the plugin.

As this plugin is just a thin wrapper around docker-compose, I'm afraid we are not able to fix it until docker-compose supports this feature.

[menulis]

Thanks for the explanation!

In the meantime, wouldn't it be possible to capture which networks are created by every docker-compose up invocation as part of composeUp and then make sure all of them are removed as the very last step of composeDown? Alternativelly, those networks could be identified by inspecting all docker networks and filtering those that have matching com.docker.compose.network and com.docker.compose.project labels.

[augi]

It could be possible but I feel it would be too hacky 😢 E.g. we should check which service uses which networks and remove just the networks that are used exclusively by the startedServices. And maybe there would be more difficulties.

[menulis]

I agree that it may not be 100% clean from the docker-compose perspective, but docker network create does accept labels that later can be used as a filter for the docker network prune command. See also https://github.com/docker/compose/issues/6636.

[augi]

If you believe it is doable (and not too hacky), please feel free to prepare a PR 🙏