[BUG] Port Mappings stopped working.

[cocoonkid]

Description

Example docker compose file :

 postgres-db:
    <<: *default-app
    image: postgres:15.4
    command: "postgres -c 'max_connections=1000' -B 2048MB  -c config_file=/pg_data/postgresql.conf"
    ports:
      - "0.0.0.0:5432:5432/tcp"
      - "10.254.66.2:5432:5432/tcp"
    healthcheck:
      test: ["CMD", "pg_isready", "-q", "-d", "acl", "-U", "postgres"]
      timeout: 10s
      interval: 5s
      retries: 3

Now I get:

Error response from daemon: driver failed programming external connectivity on endpoint postgres-db-1 (de5273980b69fbf9294910457e7acaed86f39d91e2d56ee1a4a5c61737384cc6): Error starting userland proxy: listen tcp4 0.0.0.0:5432: bind: address already in use

This worked for weeks. I tested this on two fresh installed machiness. Same behaviour. The interfaces exist.

Did something change in compose ?

I am not able to find a solution after spending 2 days on it. Usually I am pretty good at docker debugging but I am reaching my limit here :-)

Steps To Reproduce

Set two interfaces with same port in a docker compose file.

    ports:
      - "0.0.0.0:5432:5432/tcp"
      - "10.254.66.2:5432:5432/tcp"

Expected : Should work Result: Error response from daemon: driver failed programming external connectivity on endpoint marvin-postgres-db-1 (de5273980b69fbf9294910457e7acaed86f39d91e2d56ee1a4a5c61737384cc6): Error starting userland proxy: listen tcp4 0.0.0.0:5432: bind: address already in use

output of lsof and ss- > there is clearly nothing running. Host postgres has been disabled ages ago and verified it is still in this state.

❯ sudo ss -lptn 'sport = :5432'
State                     Recv-Q                     Send-Q                                          Local Address:Port                                           Peer Address:Port                     Process

❯ ss -tulpn
Netid              State               Recv-Q              Send-Q                           Local Address:Port                              Peer Address:Port              Process
udp                UNCONN              0                   0                                      0.0.0.0:51824                                  0.0.0.0:*
udp                UNCONN              0                   0                                      0.0.0.0:5353                                   0.0.0.0:*
udp                UNCONN              0                   0                                      0.0.0.0:40891                                  0.0.0.0:*
udp                UNCONN              0                   0                                    127.0.0.1:323                                    0.0.0.0:*
udp                UNCONN              0                   0                                         [::]:51824                                     [::]:*
udp                UNCONN              0                   0                                         [::]:5353                                      [::]:*
udp                UNCONN              0                   0                                        [::1]:323                                       [::]:*
udp                UNCONN              0                   0                                         [::]:57701                                     [::]:*
tcp                LISTEN              0                   128                                    0.0.0.0:8022                                   0.0.0.0:*
tcp                LISTEN              0                   4096                                   0.0.0.0:8001                                   0.0.0.0:*
tcp                LISTEN              0                   511                                  127.0.0.1:38239                                  0.0.0.0:*                  users:(("node",pid=13430,fd=19))
tcp                LISTEN              0                   4096                                   0.0.0.0:1883                                   0.0.0.0:*
tcp                LISTEN              0                   4096                                 127.0.0.1:631                                    0.0.0.0:*
tcp                LISTEN              0                   4096                                   0.0.0.0:1050                                   0.0.0.0:*
tcp                LISTEN              0                   1024                                 127.0.0.1:33105                                  0.0.0.0:*                  users:(("code-019f4d1419",pid=14931,fd=9))
tcp                LISTEN              0                   4096                                     [::1]:631                                       [::]:*
tcp                LISTEN              0                   128                                       [::]:8022                                      [::]:*
tcp                LISTEN              0                   4096                                         *:51824                                        *:*

Compose Version

Docker Compose version v2.24.6

Docker Environment

Client: Docker Engine - Community
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.6
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 9
  Running: 6
  Paused: 0
  Stopped: 3
 Images: 12
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-362.18.1.el9_3.0.1.x86_64
 Operating System: Rocky Linux 9.3 (Blue Onyx)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.13GiB
 Name: nuc-dev-1
 ID: 6605c644-0d81-43a6-b2c4-fbd66a36b07e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

I also tried the same on ubuntu 22.04 with the same non-working result.

Anything else?

ports:
  - "127.0.0.1:5432:5432/tcp"
  - "10.254.66.2:5432:5432/tcp"
  
  This works!   But it is not ideal. 
  Sometimes I want to expose the  container to the local network so anybody can reach it.
  
  

[luciangabor]

Out of curiosity, are you sure that worked? Was "10.254.66.2:5432:5432/tcp" necessary? Isn't binding on 0.0.0.0 enough?

Binding with the same port on 0.0.0.0 and another ip fails without compose (this doesn't look like a compose bug):

$ docker run --rm -p "0.0.0.0:8080:8080" -p "127.0.0.1:8080:8080" hello-world
docker: Error response from daemon: driver failed programming external connectivity on endpoint awesome_bhaskara (d7ec2c6a6959588eb6fa3c63d60d3d65c9d1e214b634fe10539e47144dc26c97): Error starting userland proxy: listen tcp4 0.0.0.0:8080: bind: address already in use.

$ docker run --rm -p "127.0.0.1:8080:8080" -p "0.0.0.0:8080:8080" hello-world
docker: Error response from daemon: driver failed programming external connectivity on endpoint blissful_stonebraker (4fe64bfdfbe0586df9392ef480eb60f283baebdb5d263598199ac948f6af3b52): Error starting userland proxy: listen tcp4 127.0.0.1:8080: bind: address already in use.

[cocoonkid]

Not strictly necessary but being explicit makes clear which other networks are involved. Especially for others as this manifest is under svc. Explicit is better than implicit :-)

Also when binding 0.0.0.0 I open it up to the public. While with a dedicated interface IP it can only be reached on that interface.

Yes it fails now.

But I am 100% sure this worked over the last few months.

I only found this issue because I stopped the containers by accident and when I restarted them this problem appeared.

[luciangabor]

Explicit is better than implicit :-)

Also when binding 0.0.0.0 I open it up to the public. While with a dedicated interface IP it can only be reached on that interface.

by binding to 0.0.0.0 you're being very explicit - you're binding on all available interfaces

as far as I know, it's either-or (and it has always been like that): you're either binding on all interfaces or you bind on specific interfaces, case when you need to be specific about their IPs (and really "have" them, but 0.0.0.0 is not such an address; maybe 10.254.66.2 was supposed to be your LAN address ...)

[cocoonkid]

I mean being explicit in the manifest so anyone immediately understands the setup.

10.254.66.2 is a wireguard interface.

And I think 0.0.0.0 and then mentioning 10.254.66.2 as well makes a lot of sense.

That way I do not have to know the specific LAN or WIFI interface that the machines connects to. But I can always make sure it will also listen on the CORRECT wg interface.

And again, it worked the last few months. It's in my git history like that.

I built my ansible automations as well like that. So it would be weird if this changed now and I have to work around this.

[NorCoker]

Ubuntu 22.04 and same docker and docker compose version - same problem on a machine where the host have multiple ip addresses on one interface - port mapping only works on the first defined docker network via ports !

docker-compose.yaml is not changed since months

[milas]

We just released Compose v2.24.7, which includes several networking fixes that I believe will address this. It's available now via GitHub releases, and will be in the apt/rpm repos with Docker Engine 25.0.4 soon.

Let us know if you still see this issue after upgrading!

[NorCoker]

Thanks to the fix ! My problems are fixed with it :-)

[cocoonkid]

Updated but unfortunately the problem persists.

I will properly report soon. Probably it's now something wrong in my setup.

[andrew-aladjev]

Confirmed this issue on Ubuntu 24.04 with docker-compose-v2 version 2.24.6+ds1-0ubuntu2. The problem for me is the following:

entrypoint: ./wstunnel server --restrict-to "localhost:${UDP_PORT}" ...
ports:
  - "${UDP_PORT}:${UDP_PORT}/udp"

I can't pass local wireguard udp port to wstunnel container. I can't replace localhost with some other host, because wstunnel uses this host for some validation purposes.

The workaround is to drop udp suffix, you can just use - "${UDP_PORT}:${UDP_PORT}".

Another solution is to use host network instead of doing port mapping. This solution is more stable and simple.