[BUG] Overlay network not found on worker node

[hibare]

Description

Docker Engine is running in swarm mode. I manually created an overlay attachable network foo on the manager node. Docker Compose services on the manager node start and attach to the network successfully. Docker Compose services on the worker node cannot find the network.

Network on manager node

$ docker network ls
NETWORK ID     NAME                 DRIVER    SCOPE
7191e5026ba1   bridge               bridge    local
qk501ogrjzoy   db                   overlay   swarm
97cc826964ef   docker_gwbridge      bridge    local
yr5u2xom773h   external             overlay   swarm
8cfe895472d9   host                 host      local
dss0u1vu52nr   ingress              overlay   swarm
e0919cac98d3   none                 null      local
ozajdr83ello   secure               overlay   swarm

docker compose up on worker node

[+] Building 0.0s (0/0)                                                                                         
network secure declared as external, but could not be found

Steps To Reproduce

1. Install latest version of docker & docker-compose as of today 2. create an attachable overlay network on manager node 3. on worker node, create a docker-compose yml file with definition to previously created overlay network 4. run docker compose up

Compose Version

v2.24.1

Docker Environment

Client: Docker Engine - Community
 Version:    25.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 9
 Server Version: 25.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: 8atlxcnaa137ogm2pdhn14skl
  Is Manager: false
  Node Address: 100.93.55.105
  Manager Addresses:
   100.102.93.57:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-1049-oracle
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 952.2MiB
 Name: in-oci-02
 ID: I3BA:GQTD:MBCM:AAWG:2S5W:NFQ4:JKEG:WSCT:GM5K:PNHF:RZSF:IGLS
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[hibare]

Downgraded to 2.21.0, everything works as expected.

For anyone who is having the same issue and want to downgrade.

1. List previous versions

sudo apt list -a docker-compose-plugin

Example ouput:

docker-compose-plugin/jammy 2.24.1-1~ubuntu.22.04~jammy arm64 [upgradable from: 2.21.0-1~ubuntu.22.04~jammy]
docker-compose-plugin/jammy,now 2.21.0-1~ubuntu.22.04~jammy arm64 [installed,upgradable to: 2.24.1-1~ubuntu.22.04~jammy]
docker-compose-plugin/jammy 2.20.2-1~ubuntu.22.04~jammy arm64
docker-compose-plugin/jammy 2.19.1-1~ubuntu.22.04~jammy arm64
docker-compose-plugin/jammy 2.18.1-1~ubuntu.22.04~jammy arm64
docker-compose-plugin/jammy 2.17.3-1~ubuntu.22.04~jammy arm64
docker-compose-plugin/jammy 2.17.2-1~ubuntu.22.04~jammy arm64
docker-compose-plugin/jammy 2.16.0-1~ubuntu.22.04~jammy arm64
docker-compose-plugin/jammy 2.15.1-1~ubuntu.22.04~jammy arm64
docker-compose-plugin/jammy 2.14.1~ubuntu-jammy arm64
docker-compose-plugin/jammy 2.12.2~ubuntu-jammy arm64
docker-compose-plugin/jammy 2.12.0~ubuntu-jammy arm64
docker-compose-plugin/jammy 2.11.2~ubuntu-jammy arm64
docker-compose-plugin/jammy 2.10.2~ubuntu-jammy arm64
docker-compose-plugin/jammy 2.6.0~ubuntu-jammy arm64
docker-compose-plugin/jammy 2.5.0~ubuntu-jammy arm64
docker-compose-plugin/jammy 2.3.3~ubuntu-jammy arm64

2. Install previous version

sudo apt install docker-compose-plugin=2.21.0-1~ubuntu.22.04~jammy

[ndeloof]

AFAIK networks created on a swarm cluster are only created on worker node "on first usage" by a swarm service. As compose does not integrate with swarm, it require such a network to exist, and won't trigger swarm to create on-demand.

so basically, don't try to use compose with swarm. Better use swarm on its own

[hibare]

This is something new. The previous versions of docker compose just works fine.

Here is a similar issue that I had opened https://github.com/docker/compose/issues/10588

Query: is this the same with overlay network which are attachable?

Correct me if I misunderstood something.

[ndeloof]

The code actually ignores checking for network existence as long as it is declared both external and with driver: overlay. IIRC we wanted here to avoid not detecting a missing network, while still being able to support overlay networks created for swarm. This double check probably doesn't apply to your compose file. Can you please confirm you have an external overlay network declared without driver attribute ?

As noted in code, it doesn't make sense to me user has to define config for a network that is flagged as external. We should relax this rule.

[hibare]

No, all overlay networks are defined with external and driver: overlay.

I'll test without driver and update here.

[ndeloof]

I don't understand what's wrong here then :'(

[kaanrkaan]

Same issue here

On master node:

NETWORK ID     NAME              DRIVER    SCOPE
fd67d6cae42f   bridge            bridge    local
ed357627454a   docker_gwbridge   bridge    local
0429a296e264   host              host      local
35msgwq0hd7j   ingress           overlay   swarm
bf52bb855861   none              null      local
6ga4tu2iqldd   overnet           overlay   swarm

On worker node docker compose:

networks:
  overnet:
    external: true
    driver: overlay

Error response from daemon: network overnet not found

After I run

docker run -dit --name alpine2 --network overnet alpine It configures the network from the swarm. Thus, able to run the same docker compose file

Both worker and master node is running on the Docker version 25.0.1, build 29cf629

• Ps: same issue occurs with this compose as well

networks:
  overnet:
    external: true

[jojolll]

Hello,

On version 2.24.5, where the fix seems to have been released (since 2.24.4), it still doesn't work. I have to create a container with

docker run -dit --name alpine1 --network my_overlay alpine to get the worker network created and the compose running (same as kaanrkaan).

[jojolll]

Hello,

Problem still present in v2.24.6

@kaanrkaan have you tested the latest versions? Do you confirm these findings?

I guess the fix doesn't work as expected. Should I open another issue?