moby
[Feature Request] Bind a specific network interface for a particular overlay network
Hi, I posted this issue in moby/moby but I think this issue relates more to this repo.
I want to bind a specific network interface for my user-defined overlay network. The reason for this is that DigitalOcean (as well as other cloud providers) offer private networking which allows you to communicate to other VMs within the same data center (region) without going over the public internet and being charged.
If I'm correct, the network interface that each overlay network created on a node goes over is specified when you join the swarm with --advertise-addr. Is it possible to somehow use this as a default, but in addition, when we create a new overlay network we can explicitly specify the network interface we want to use?
ie. docker network create -d overlay --iface eth1 myoverlaynetwork or docker network create -d overlay --internal-ip PRIVATE_IP myoverlaynetwork
In my opinion, this is not even an ideal approach to the problem because not every distribution names their network interfaces with ethX, let alone we cannot guarantee that every node will have the private network interface to be eth1. It could be the case that when a VM is started, it creates eth0 and eth1, but the public interface is actually eth1.
Ultimately, I want to define these overlay networks in a service stack config so I can docker stack deploy my services so that they go over the private network instead of the public internet. I have several overlay networks to isolate container communication, some of which goes across different datacenters, and others within the same datacenter. The services that are running in the same region/datacenter, however, should communicate through the private network interface instead of the public network interface which I am trying to do via another overlay network.
I've also looked into other using other drivers (macvlan, bridge, etc), but I'm not entirely sure if it's possible to do what I want to accomplish.
I was able to get a setup working with ipvlan with a different cloud provider, Vultr, but unfortunately, it does not work with every cloud provider; in particular, DigitalOcean. The private network that they setup for you is actually a shared private network, so trying to create containers with ip addresses within the same subnet will not work (trying to ping a container to another container in different hosts will not see the ARP request, let alone reply).
So, I propose creating an overlay network with a specified network interface by doing something similar to what we need to do with ipvlan/macvlan in swarm mode, which is creating a config-only network on the nodes, and on a manager, we create the overlay network itself which will distribute across the swarm using the --config-from flag.
Example:
Worker1: docker network create --config-only -o iface=eth1 myconfig
Worker2: docker network create --config-only -o iface=ens7 myconfig
Manager: docker network create --driver overlay --config-from myconfig mynetwork
And in the stack file, we simply set mynetwork to external, ie:
networks:
mynetwork:
external: true
services:
myservice:
networks:
- mynetwork
I believe implementing this should not be too difficult, and it also addresses the issue of not every Linux distribution naming their interface ethX.
Also related to https://github.com/moby/moby/issues/24317 (which is tracking this in Moby)
