Network cleanup not always working

[vmassol]

Module

Core

Testcontainers version

1.17.3

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host Arch

x86

Docker version

$ sudo docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:37 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:46 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

We're getting regularly the error Caused by: com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network"} in our TC-based tests.

While researching this, I've found that several networks created by TC remain and are not deleted after the tests. For example on our CI agent:

$ sudo docker network ls
NETWORK ID     NAME                                   DRIVER    SCOPE
fc00e553e083   40fde7d3-324c-42cc-aeac-bfa9cd68cd0d   bridge    local
0a2f5e0ebc2f   47c53062-6c06-415d-a961-b45cf3f686af   bridge    local
2b7f4773ec7b   81ea6de2-0237-4eb7-b5c3-ed545f34d33c   bridge    local
1a56194c0416   38864bbb-de57-4cb3-9fd2-759056daa61c   bridge    local
d5d4b01c9f0a   41787e72-4ba5-43c2-a83f-6bf6949ca4c1   bridge    local
62ae2d1e22f2   52294b16-867d-4a66-930a-3e2432c898f1   bridge    local
79bd7f2e9a9a   2628661f-de18-45a8-80bd-f13c7a8f5aa1   bridge    local
e2072e729583   a7b7b131-15f4-4dd1-aa7e-35f548f4b7fd   bridge    local
13ecf70ca868   b29b8b18-9c8a-48cb-b628-763f936a12bc   bridge    local
88216c595688   bridge                                 bridge    local
623732a6b674   c864c52c-f630-42e6-a123-d0ad590063a5   bridge    local
08640b19df0f   ebaca507-973e-4ba5-be7f-d01987344b3f   bridge    local
4bcdbd714e78   f9111034-9c6f-4e67-9ce7-89f2719ee689   bridge    local
3f318d295a89   host                                   host      local
c3aad684575a   none                                   null      local

If I inspect the networks with the numbers in their name, I can see they're labelled as TC ones:

$ sudo docker  network inspect e2072e729583
[
    {
        "Name": "a7b7b131-15f4-4dd1-aa7e-35f548f4b7fd",
        "Id": "e2072e729583dd48d347dc333dc7a0f05f5627d93f3a4514cf25a2c76cfced00",
        "Created": "2022-07-20T04:22:05.82718144Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.21.0.0/16",
                    "Gateway": "172.21.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {
            "org.testcontainers": "true",
            "org.testcontainers.sessionId": "91cbf7fa-3a02-4279-94dd-c59397225ce1"
        }
    }
]

Is this a known bug/limitation? Is there something I can do about it?

Thanks!

Relevant log output

No response

Additional Information

No response

[bsideup]

@vmassol do you happen to have Ryuk disabled?

[vmassol]

@bsideup nope, we love Ryuk and we never disable it ;)

[vmassol]

Also, most of the time the networks are cleaned fine (judging by the amount of tests we do every day and the number of stray docker networks remaining).

[Git-idiot]

We faced the same problem