Proposal: provide the ability to black list IP address ranges

[mixja]

This is to address the common issue of Docker environments needing to communicate with other private networks and avoiding IP addressing conflicts.

The current IPAM driver allows you to specify IP subnets, etc to use for Docker networks, however in a continuous delivery use case where agents running Docker may spin up many different multi container Docker environments, it is difficult to use IPAM for this use case (you end up having to trying to ensure unique IP address ranges for each environment that may run on a single Docker Host - in the event you have different environments attempting to use the same IP subnet, you get failures).

For this scenario, it would be much better to be able to black list specific IP address ranges (to ensure connectivity to other private networks) and allow Docker to choose an appropriate IP address range from the remaining "white listed" address space.

[nerdalert]

Hi @mixja have you ever tried --aux-address=x.x.x.x (lets you exclude any address from being handed out) and --ip-range (lets you allocate a smaller pool within the larger IPAM pool for a network) options? I just added an example using both of them in a network create and thought it might be what you are looking for if I am not misunderstanding your comment. The commands are compatible using the various drivers, bridge, overlay, etc, not just the Macvlan/Ipvlan drivers since they all leverage the default IPAM plugin from Libnetwork core.

/Snipped from: Macvlan Driver https://github.com/docker/libnetwork/blob/master/docs/macvlan.md

While the eth0 interface does not need to have an IP address, it is not uncommon to have an IP address on the interface. Addresses can be excluded from getting an address from the default built in IPAM by using the --aux-address=x.x.x.x argument. This will blacklist the specified address from being handed out to containers from the built-in Libnetwork IPAM.

The next example demonstrates how default gateways are inferred if the --gateway option is not specified for a subnet in the docker network create ... command. If the gateway is not specified, the first usable address in the subnet is selected. It also demonstrates how --ip-range and --aux-address are used in conjunction to exclude address assignments within a network and reserve sub-pools of usable addresses within a network's subnet. All traffic is untagged since eth0 is used rather then a sub-interface.

docker network create -d macvlan \
  --subnet=192.168.136.0/24 \
  --subnet=192.168.138.0/24 \
  --ipv6 --subnet=fd11::/64 \
  --ip-range=192.168.136.0/25 \
  --ip-range=192.168.138.0/25 \
  --aux-address="reserved1=fd11::2" \
  --aux-address="reserved2=192.168.136.2" \
  --aux-address="reserved3=192.168.138.2" \
  -o parent=eth0 mcv0

docker run --net=mcv0 -it --rm alpine /bin/sh

Next is the output from a running container provisioned on the example network named mcv0.

# Container eth0 output (the fe80::42:c0ff:fea8:8803/64 address is the local link addr)
ip address show eth0
100: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 02:42:c0:a8:88:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.136.3/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd11::3/64 scope global flags 02
       valid_lft forever preferred_lft forever
    inet6 fe80::42:c0ff:fea8:8803/64 scope link
       valid_lft forever preferred_lft forever

# IPv4 routing table from within the container
$ ip route
default via 192.168.136.1 dev eth0
192.168.136.0/24 dev eth0  src 192.168.136.3

# IPv6 routing table from within the container (the second v6 addresses is the local link addr)
$ ip -6 route
fd11::/64 dev eth0  metric 256
fe80::/64 dev eth0  metric 256
default via fd11::1 dev eth0  metric 1024

[jmenga]

The problem here is about communication with other internal networks from a Docker container. If Docker chooses an IP subnet that is in use elsewhere in the network, containers on the Docker network won't be able to communicate with the other internal network.

So if I have an existing internal 172.17.0.0/16 network and Docker Compose chooses that range when it's creating a network, I'm out of luck. Yes I can explicitly define an IP range via IPAM that doesn't conflict, but what if I have a lot of Docker Compose environments. I now need to manage IP addressing for each Docker Compose environment (to ensure they don't conflict with other networks or other Docker Compose networks on a given host) which is a complete anti-pattern in this new world.

The --ip-range option is no good, as Docker still creates a network that encompasses the defined IPAM subnet and any other Docker Compose environment running on the same Docker host must avoid that network. The --aux-address option is more of a DHCP exclusion parameter and isn't really relevant.

Ideally, if I want to be able to define an IP addressing block or set of blocks, from which Docker Compose can either pick a random subnet that's not in use locally (i.e. a whitelist), or completely avoid choosing subnets from the specified block(s) (i.e. a blacklist).

[HadesArchitect]

@jmenga Do you have any workaround to avoid the issue? We run into it again and again. =/

[jmenga]

@HadesArchitect - ultimately I settled on creating an external network using the local driver.

The advantages of this approach:

• You can set a safe range
• Docker DNS is smart enough to make this work even if you have different Docker Compose projects running at the same time with the same service names. So the app and db services for Project A can still resolve each others names correctly, and the app and db services for Project B can still resolve each others names correctly, even though they are on the same network and running at the same time
• You can selectively expose services in each project with external DNS names - this could be useful for more complex meta-project scenarios where a service from Project A may want to talk to a service in Project B

The disadvantages:

• You have to create and manage the lifecycle of the external network yourself

In our workflows, we use the Make build system so we created a make init task that is always called and ensures the external docker network is created.

Example docker-compose.yml file:

version: '2'

# Example of using an external network as default network
# This example relies on environment variable NETWORK_NAME
networks:
  default:
    external: 
      name: ${NETWORK_NAME}

# Example of exposing the 'app' service with an external DNS name 
# of 'project-a-app' to other Docker Compose projects
services:
  app:
    image: myorg/nginx
    networks:
      default:
        aliases:
          - project-a-app

Example Makefile:

# The export keyword exports this setting into all subshells that Make spawns
export NETWORK_NAME := cd_workflow
NETWORK_SUBNET := 192.168.200.0/24
NETWORK_GW := 192.168.200.1
NETWORK_ID := $(shell docker network ls -f name=$(NETWORK_NAME) -q)

# Creates workflow infrastructure such as external networks and volumes
init:
  @ $(if $(NETWORK_ID),,docker network create --subnet=$(NETWORK_SUBNET) --gateway=$(NETWORK_GW) $(NETWORK_NAME))

# Other tasks call init first to ensure external network is created
test: init
  ...
  ... 

[GordonTheTurtle]

@mixja It has been detected that this issue has not received any activity in over 6 months. Can you please let us know if it is still relevant:

• For a bug: do you still experience the issue with the latest version?
• For a feature request: was your request appropriately answered in a later version?

Thank you! This issue will be automatically closed in 1 week unless it is commented on. For more information please refer to https://github.com/docker/libnetwork/issues/1926

[mixja]

@GordonTheTurtle yes this issue is still present and relevant.

You should be able to define blacklists that avoid IP addressing conflicts with your network, and still allow you to let Docker dynamically choose an available IP address range.

[Vacant0mens]

@nerdalert @GordonTheTurtle

I'm seeing this problem come up in Windows as well.

I posted this issue a while ago about it. We use docker stack deploy with a compose file. We've been allowing docker to automagically create the stack/overlay networks in the swarm, but they've been being created using a similar/included subnet as our local network and it's been causing problems. (i.e. our local subnet is 10.0.0.0/16, docker creates 10.0.0.0/24) we have been seeing a lot of dropped requests and and lost messages because of it.

Why doesn't docker/libnetwork just exclude the host's network subnets all the time? Seems like if you're going to create a virtual network automagically (especially for overlay/ingress) you would want it to have its own routing and avoid network conflicts, ergo, exclude the host's network by default. Not excluding the host's network is just asking for all sorts of network conflicts.

Is there a way to use a compose file to exclude subnets when creating the stack network rather than having to create the network before deploying the stack?

[Vacant0mens]

@mixja @HadesArchitect

Here's a Workaround:

Create a dummy network in Docker that covers the hosts' local subnet, but then don't use it for anything.

i.e. if your local network is 10.10.0.0/24, create a network in the swarm like this:

docker network create -d overlay --subnet 10.10.0.0/24 xx_local_reserve_do_not_use_xx

Then when you go to deploy your stack or swarm services, or create other networks for your services, it won't pick up that address range since it's being used.

Pretty sketchy, but it seems to be working for me.

[selansen]

One can use above workaround or we have new feature which will allow user to select subnet ranges for their docker network. https://github.com/docker/libnetwork/pull/2058. This should address your issue indirectly because this PR will give control to you to choose subnet ranges. Also there are few ways to ignore if you want to skip only specific IP address instead of subnet range.

[selansen]

Quick update : #2058 supports only local bride networks. Its not for overlay networks

[Vacant0mens]

@selansen is there a feature update in the works that will address this conflict automatically? checking host network settings and pulling IP ranges before queuing up the creation of a new network doesn't seem like a complicated process. I mean, libnetwork is already checking Docker's network subnets before it auto-creates a new network, right? Would it be too much work to check the host's network at the same time?

[Bengrunt]

@Vacant0mens This would be a start but this will not solve the site wide routing issue.

You may have a subnet used somewhere else in your organization (i.e. 172.17.x.x) but not on the docker host. The routing issues will remain because different hosts in your network will be routing the same IP ranges to different locations.

So definitely, a blacklist would be nice! :)

[Vacant0mens]

@Bengrunt Yes, I agree that it would be near impossible to be able to detect any usage of subnets throughout an entire infrastructure, and blacklisting would be a solid fix for that problem. But the bigger problem is that Docker doesn't even attempt to exclude any networks by default, not even the subnet that the host is directly attached to. (see my first comment in this issue)

I personally have seen it cause routing issues for applications/containers attempting to resolve addresses both inside and outside of the overlay network at different times causing dropped requests both client-side and application-side.

It baffles me that this isn't already a thing.

[selansen]

as of now we are not having plan to add new command specifically blacklisting subnet ranges. Will get back to you all soon.

[Vacant0mens]

@selansen is there another ticket/issue open for creating a feature to exclude the host's subnet when creating new networks? Is anyone at Docker aware of this issue?

[selansen]

Have asked about it to @mark-church . Will get back to you.

[Vacant0mens]

@selansen @mark-church Any word yet?

[selansen]

I will talk to @mark-chruch and update you soon.

On Thu, Aug 16, 2018 at 11:13 AM, Vacant0mens [email protected] wrote:

@selansen https://github.com/selansen @mark-church https://github.com/mark-church Any word yet?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/docker/libnetwork/issues/1167#issuecomment-413580665, or mute the thread https://github.com/notifications/unsubscribe-auth/AKEKn2MU42GwRWOQXZlTJH3rB_ua4hw3ks5uRYwrgaJpZM4IdC-_ .

[mark-church]

@Vacant0mens please see https://github.com/docker/libnetwork/pull/2241

It is Default Address Pools and instead of blacklisting addresses it is a whitelist model where multiple discontiguous pools can be configured such that subnets will be taken from the pools sequentially. A global default subnet size (that will apply to all subnets taken from the pool) is also a part of this configuration.

[Vacant0mens]

Does this model also include excluding the host's local subnet?

[mark-church]

No it's an inclusive model and will include whatever you set it to so if you set the default address pool to 10.10.0.0/16 then overlay subnets will be taken from this range. This PR is specific to global (overlay) networks in swarm.

[Vacant0mens]

Why is that not the default? If someone doesn't know that, it can cause all sorts of problems.

[selansen]

@Vacant0mens , I didnt understand your question "Why is that not the default?". we have inbuilt default address pool for overlay as well bridge network. now we are enabling users to set the default pool .

[ctelfer]

I believe he is asking why it wasn't the default behavior that the address pool ranges for local and overlay networks be configurable, noting that making assumptions about which addresses are available can conflict with one's local network config. I guess the only reasonable answer to that question is that, until recently, it hasn't been a common enough problem for people to request or offer fixes for it.

[archfz]

Up. We have also encountered this issue. We have multiple projects with docker and some people from a specific range of local IP cannot connect to the VM which hosts multiple docker networks since there is a route with the same IP range.

[prologic]

The stated "dummy network" work-around also worked for me. I spent days trying to work out wtf was going on with my mail server (running poste) as a Docker Swarm service; only to discover that the mail network I created externally via docker network create -d overlay mail some time ago was colliding with my internal private LAN and causing monitoring to fail. It took me to resort to tcpdump'ing to discover this the hard way :)

It would be nice to configure the Docker engine in this case to blacklist internal CIDRs that would cause network collisions like this.

[Vacant0mens]

@ctelfer @selansen @mark-church My question was short, so let me explain a little. If an overlay network is created that matches the host's subnet, the environment will have network conflicts.

As brought up by @prologic, myself, and many others, we are not the first people, nor will we be the last, to waste days trying to figure out that as the end-user we were somehow magically supposed to know that we need to make a dummy network that we don't ever use or touch so that our swarm-based service containers can talk to our local subnet properly. (side note: docker network prune removes this dummy network, so this issue could potentially come up multiple times for the same person/team)

So my question was: why is it not already default to exclude (or "not include", since we're talking about a whitelist schema) the host's external subnet? I'm not asking for a blacklist, I'm asking for default functionality that should have been part of the original design (see my first posted issue on this subject here and another one here )

The swarm handles a good amount of data about each node, including what address and subnet they're attached to. It wouldn't be a hard function to write to say "create overlay network 10.0.0.0/24 ... oh wait, node xyz has address 10.0.0.145/24, better try another. ... create network 10.0.1.0/24 ... oh wait, node abc has address 10.0.1.25/24, better try another."

I realize that this might be a slight over-simplification of the code that needs to be written, but this should have been part of the design from the beginning because (like I said) If an overlay network is created that matches the host's subnet, the environment will have network conflicts.