compose
compose copied to clipboard
docker
docker-compose V2 fails for port range and replicas>1 where V1 would succeed
Description
Using multiple replicas of a container where each replica should use one of the ports selected by a port-range:
version: "3.4"
services:
dummy:
image: busybox
command: ["sh", "-c", "while true ; do sleep 3 ; date ; done"]
deploy:
replicas: 2
ports:
- "6005-6015:5000"
When starting up these containers with docker-compose V1 the container instances use port-mapping where each container picks one host-port from the selected port range to map to internal container port.
$ docker-compose ps
Name Command State Ports
----------------------------------------------------------------------------------------------------------------
docker-compose-issue_dummy_1 sh -c while true ; do slee ... Up 0.0.0.0:6007->5000/tcp,:::6007->5000/tcp
docker-compose-issue_dummy_2 sh -c while true ; do slee ... Up 0.0.0.0:6008->5000/tcp,:::6008->5000/tcp
When starting up these containers with docker-compose V2 the containers use all ports from the selected port range to map to internal container port, resulting in error like the following for the second instance of that service:
Error response from daemon: driver failed programming external connectivity on endpoint docker-compose-issue_dummy_1 (6d9cff4937b3b6946e0bb949a36362cf91477f841be34ac9bd2693453b6e8837): Bind for 0.0.0.0:6009 failed: port is already allocated
Steps to reproduce the issue:
- Switch to docker-compose V1,
docker-compose disable-v2 - Startup containers,
docker-compose up -d - List running containers and the port mapping they use,
docker-compose ps - Shutdown service,
docker-compose down - Switch to docker-compose V2,
docker-compose enable-v2 - Startup containers,
docker-compose up -d - docker-compose fails to startup because of port mapping clash
Describe the results you received:
docker-compose.yaml only seems to work for V1 and not V2.
Describe the results you expected:
I would expect that docker-compose.yaml can be run by V1 and V2 alike.
Output of docker version:
Client:
Cloud integration: 1.0.17
Version: 20.10.7
API version: 1.41
Go version: go1.16.4
Git commit: f0df350
Built: Wed Jun 2 11:56:22 2021
OS/Arch: darwin/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.7
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: b0f5bc3
Built: Wed Jun 2 11:54:58 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.6
GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc:
Version: 1.0.0-rc95
GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker info:
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
compose: Docker Compose (Docker Inc., 2.0.0-beta.3)
scan: Docker Scan (Docker Inc., v0.8.0)
Server:
Containers: 30
Running: 10
Paused: 0
Stopped: 20
Images: 136
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.25-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 3
Total Memory: 11.7GiB
Name: docker-desktop
ID: 65BF:GFI6:S3ZY:URLI:EHO5:GFN5:PT7Y:Q7JW:ACRV:FJCE:YM5X:2YFM
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 159
Goroutines: 157
System Time: 2021-06-24T14:52:06.2642643Z
EventsListeners: 5
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS ECS, Azure ACI, local, etc.):
local, MacOS BigSur 11.4
When running the example from above the V1 docker-compose will issue a warning but proceed without an error:
WARNING: The "dummy" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Is it possible that V2 is more restricted and hence just fails instead of reporting a warning like V1 ?
Additionally the docker-compose ports documentation is not very clear about this apparently working port-range feature in V1, i.e. automatically assign a different port from the given range of ports to multi-replica-services.
V2 indeed doesn't reproduce this V1 warning, this is something we should consider.
documentation is not very clear about this apparently working port-range feature in V1 Indeed, I didn't know about it :D
Compose v1 creates containers sequentially, as an equivalent to docker run. Then the port specification is passed to engine. According to https://docs.docker.com/engine/reference/run/#expose-incoming-ports:
"When specifying a range for hostPort only, the
containerPort must not be a range. In this case the
container port is published somewhere within the
specified hostPort range. (e.g., -p 1234-1236:1234/tcp)".
Compose v2 creates containers in parallel, so this engine feature doesn't apply and this results in a port conflict. Seems we will have to drop this (premature?) optimization to restore this feature.
Hi
I have same problem on V2. The host ports are assigned to one container instead of distributing them among all replicas.
d0ac8e9625f3 bitnami/redis-sentinel:latest "/opt/bitnami/script…" About a minute ago Up 53 seconds 0.0.0.0:26379->26379/tcp, :::26379->26379/tcp, 0.0.0.0:26380->26379/tcp, 0.0.0.0:26381->26379/tcp, :::26380->26379/tcp, :::26381->26379/tcp os24_support_redis-sentinel_3
On V1 replicas works just fine:
1b5d5d7fbe67 bitnami/redis-sentinel:latest "/opt/bitnami/script…" 7 seconds ago Up 3 seconds 0.0.0.0:26379->26379/tcp, :::26379->26379/tcp os24_support_redis-sentinel_2
4fd28e9d6e21 bitnami/redis-sentinel:latest "/opt/bitnami/script…" 7 seconds ago Up 3 seconds 0.0.0.0:26381->26379/tcp, :::26381->26379/tcp os24_support_redis-sentinel_3
2c0a7016ba04 bitnami/redis-sentinel:latest "/opt/bitnami/script…" 7 seconds ago Up 3 seconds 0.0.0.0:26380->26379/tcp, :::26380->26379/tcp os24_support_redis-sentinel_1
This is still problem
Linux dc165-test 5.4.0-89-generic #100-Ubuntu SMP Fri Sep 24 14:50:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Docker version 20.10.9, build c2ea9bc
Docker Compose version v2.0.1
ran a quick test, and this actually isn't an issue with replicas. Even using replicas: 1 container is bound to the whole range, not "somrwhere within the specified range`
0.0.0.0:6005->5000/tcp, 0.0.0.0:6006->5000/tcp, 0.0.0.0:6007->5000/tcp, 0.0.0.0:6008->5000/tcp, 0.0.0.0:6009->5000/tcp, 0.0.0.0:6010->5000/tcp, 0.0.0.0:6011->5000/tcp, 0.0.0.0:6012->5000/tcp, 0.0.0.0:6013->5000/tcp, 0.0.0.0:6014->5000/tcp, 0.0.0.0:6015->5000/tcp
.. so there's something wrong with compose v2 passing port specification to the engine.
confirmed. compose-v1 creates container as:
"PortBindings": {
"5000/tcp": [
{
"HostIp": "",
"HostPort": "6005-6015"
}
]
},
compose-v2 creates container as:
"PortBindings": {
"5000/tcp": [
{
"HostIp": "",
"HostPort": "6005"
},
{
"HostIp": "",
"HostPort": "6006"
},
{
"HostIp": "",
"HostPort": "6007"
},
...
the issue is in compose-go, which "extends" 6005-6015:5000 into a list of bindings for each and every values in the range
I'm not sure if this is the same problem, but researching how to solve my own problem took me here. I was having a similar problem on a Mac,
christa % docker compose up
Recreating recorder ... error
ERROR: for recorder Cannot start service recorder: Ports are not available: listen tcp 0.0.0.0:5000: bind: address already in use
and I was able to fix it by checking:
christa % sudo lsof -i -P -n | grep 5000
ControlCe 56829 christa 22u IPv4 0xce0b919697e39c8f 0t0 TCP *:5000 (LISTEN)
ControlCe 56829 christa 23u IPv6 0xce0b919684ffd2b7 0t0 TCP *:5000 (LISTEN)
and then turning off the Airplay Receiver Service in System Preferences -> Sharing
compose-go both extends port range to a list of ports, but also assumes this is expected
and types are designed to only accept an actual port (uint32) not range.
I guess we are stuck here :'(
This will also require some clarification on https://github.com/compose-spec/compose-spec/blob/master/spec.md#ports if we want this to be addressed
- the compose spec schema does not allow use of a string as
publishedport value (https://github.com/compose-spec/compose-spec/blob/master/schema/compose-spec.json#L321)
Docker version 20.10.13, build a224086 Docker Compose version v2.3.3
I'm seeing this issue also.
@hacdias @Hazmi35 If you run the docker compose up several times (one for each replica) you can eventually get the system running with the replicas and port mappings.
Currently having to remove the replica config an repeat the container instance config for each replica I need.
This issue needs looking at...its been open for a very long time with a significant bug in the use of docker-compose when testing a scaling setup. I've moved to Kubernetes/skaffold.
$ docker --version Docker version 20.10.14, build a224086
$ docker-compose --version Docker Compose version v2.4.1
Also running into this issues. Sometimes it will bring everything up correctly, and other times I will get the bind error and one of the replicas will not start. I use Docker for Mac so I don't know if that is part of the reason.
I have a similar problem as well.
services:
api:
build: .
ports:
- 8000-8010:8000-8010
I am trying to create at least 2 instances of the same image and it fails miserably:
$ docker compose up --scale api=2
[+] Running 2/2
⠿ Container files-api-2 Created
⠿ Container files-api-1 Recreated
Attaching to files-api-1, files-api-2
Error response from daemon: driver failed programming external connectivity on endpoint files-api-2 (container id): Bind for 0.0.0.0:8010 failed: port is already allocated
I referenced this in an issue - #9727
I think this issue was resolved in the new version.
This configuration worked for me.
Docker Compose Version.
$ docker-compose -v
Docker Compose version v2.5.0
Configuration file docker-compose.yml.
version: '3.8'
services:
video:
image: video:latest
restart: always
ports:
- "4060-4062:4060"
deploy:
mode: replicated
replicas: 3
And the replicas it created:
d283b9969098 video:latest "node /usr/src/app/i…" 15 seconds ago Up 14 seconds 0.0.0.0:4062->4060/tcp, :::4062->4060/tcp video-test-video-1
e7b5075fa6c1 video:latest "node /usr/src/app/i…" 15 seconds ago Up 14 seconds 0.0.0.0:4061->4060/tcp, :::4061->4060/tcp video-test-video-3
c225ca9d71ac video:latest "node /usr/src/app/i…" 15 seconds ago Up 14 seconds 0.0.0.0:4060->4060/tcp, :::4060->4060/tcp video-test-video-2
All the replicas correctly mapped to 4060, 4061, and 4062.
It's still happening. (m1 mac, macOS 12.1 monterey)
$ docker -v
Docker version 20.10.20, build 9fdeb9c
$ docker compose version
Docker Compose version v2.12.1
services:
redis:
image: bitnami/redis:latest
environment:
- ALLOW_EMPTY_PASSWORD=yes
deploy:
mode: replicated
replicas: 3
ports:
- 6300-6400:6379
$ docker compose up -d
[+] Running 1/4
⠿ Network composetest_default Created 0.0s
⠿ Container composetest-redis-3 Starting 0.1s
⠿ Container composetest-redis-1 Starting 0.1s
⠿ Container composetest-redis-2 Starting 0.1s
Error response from daemon: Ports are not available: exposing port TCP 0.0.0.0:6300 -> 0.0.0.0:0: listen tcp 0.0.0.0:6300: bind: address already in use
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8cb371485fba bitnami/redis:latest "/opt/bitnami/script…" 8 seconds ago Up 7 seconds 0.0.0.0:6300->6379/tcp composetest-redis-3
I'd like to fix this, but don't know where to begin. Any advice?
It seems to be more of a timing issue afaik. I have seen the exact same docker-compose work if there is a delay (in my case due to high cpu load) so that the first container starts and it takes a second or two before the second container starts. When there is almost no delay (just some milliseconds) I get the bind-error.
@sh-cho Having the same issue, docker-compose v 2.6.0 on Mac M1.
Only way I could run multiple instances is by removing the port range
(e.g. I had:
ports: - "11001-11004:7777"
). I changed this to:
ports: - "7777"
and then my 4 instances could be started.
Unfortunately with random external ports, but at least they could be started.
Had exactly the same error message:
Error response from daemon: Ports are not available: exposing port TCP 0.0.0.0:11001 -> 0.0.0.0:0: listen tcp 0.0.0.0:11001: bind: address already in use
Looks like it tries to assign to "internal port 0" which is quite odd.
Cheers
AFAICT we need to make the ContainerCreate API calls sequential to enforce a distinct port is assigned
Hi, I keep facing this issue with docker compose 2.15.1 with following setting
ports:
- "8081-8082:8080"
- "8444-8445:8443"
- "9991-9992:9990"
- "10000-10001:9999"
deploy:
replicas: 2
running docker compose up starts the first instance but shows the error message (Error response from daemon: Ports are not available: exposing port TCP 0.0.0.0:8081 -> 0.0.0.0:0: listen tcp 0.0.0.0:8081: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.), running it a second time make my 2 replicas working as expected (confirmed with docker ps)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
74be71e9b594 ********************************** Up 3 minutes 0.0.0.0:8081->8080/tcp, 0.0.0.0:8444->8443/tcp, 0.0.0.0:9991->9990/tcp, 0.0.0.0:10000->9999/tcp *****************
a07e1560ee5c ********************************** Up 3 minutes 0.0.0.0:8082->8080/tcp, 0.0.0.0:8445->8443/tcp, 0.0.0.0:9992->9990/tcp, 0.0.0.0:10001->9999/tcp *****************
