Docker blocking network of existing LXC containers

[thesourcerer8]

I am running a server with LXC containers on it. I found a software that was available as Docker containers, so I installed Docker on the server and tried that software.

• [x] This is reporting unexpected problematic behaviour
• [x] This is a feature request
• [x] I searched existing issues before opening this one

Expected behavior

Both LXC containers should work normally, and new Docker containers should work. If it isn't possible for Docker to start a container without interrupting existing containers due to any conceptual issue, I would expect Docker to detect that such a problem exists and to provide an overridable error message or at least a warning, that it might interfere with existing containers.

Actual behavior

The Docker containers work, but the LXC containers are suddenly disconnected from the network. The LXC containers are still running, and I can attach to them directly, but they aren't visible on the network anymore, they cannot be pinged, and their services aren't available anymore. Docker has disconnected the running LXC containers from the network.

Steps to reproduce the behavior

Install Debian Linux Install LXC Create a LXC container with it's own IP address Run a server on the LXC container Install Docker Download and run a Docker container Try to access the server in the LXC container over the network

Output of docker version:

Client:
 Version:      17.07.0-ce
 API version:  1.31
 Go version:   go1.8.3
 Git commit:   8784753
 Built:        Tue Aug 29 17:42:54 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.07.0-ce
 API version:  1.31 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   8784753
 Built:        Tue Aug 29 17:41:44 2017
 OS/Arch:      linux/amd64
 Experimental: false

Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 4 Server Version: 17.07.0-ce Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 42 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 3addd840653146c90a254301d6c3a663c7fd6429 runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4 init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 4.9.0-3-amd64 Operating System: Debian GNU/Linux 9 (stretch) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 31.41GiB Name: servername ID: ZQY3:PKBB:X7ZY:WY54:VKG4:IILY:42LG:KURS:4MXC:JR67:A6IX:T3KT Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Http Proxy: http://proxy:8080/ Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.) Linux version 4.9.0-3-amd64 ([email protected]) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)

[aaronp24]

I ran into this yesterday and the problem seems to be how Docker messes with the FORWARD filter table in iptables:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 431K 1126M DOCKER-USER  all  --  any    any     anywhere             anywhere            
 431K 1126M DOCKER-ISOLATION  all  --  any    any     anywhere             anywhere            
 219K 1090M ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  any    docker0  anywhere             anywhere            
 211K   37M ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere   

These rules block all forwarded packets that aren't to, from, or between Docker containers. My LXC containers use a "br0" interface that's separate from the "docker0" bridge.

[ghost]

@aaronp24, how did you get it working?

[aaronp24]

I just manually flushed the FORWARD table and changed its default policy to ACCEPT

iptables -F FORWARD
iptables -P FORWARD ACCEPT

[aaronp24]

Oh wait, I forgot that I also found you could disable this behavior by putting this in /etc/docker/daemon.json:

{
	"iptables": false
}

[gmcclins]

@aaronp24 Thank you!

I can confirm that /etc/docker/daemon.json:

{
        "iptables": false
}

works. After install of docker and prior to the setting lxc ls shows no ipv4 information. After the setting I now have an ipv4 address again.

Edit: I guess I spoke a little too soon. While this does restore lxc networking, it seems docker is not able to reach the internet now. I've deleted the /etc/docker/daemon.json file I created and reloaded docker. This of course led to lxc container loosing it's address again. I then

iptables -F FORWARD iptables -P FORWARD ACCEPT

and have confirmed I now have networking in both docker and lxc.

[aaronp24]

I think this fixed it for me:

iptables -t nat -A POSTROUTING -s $dockernet -o ! docker0 -j MASQUERADE

[gmcclins]

@aaronp24

iptables -F FORWARD iptables -P FORWARD ACCEPT

fixed it for me. Should I be using

iptables -t nat -A POSTROUTING -s $dockernet -o ! docker0 -j MASQUERADE

instead?

[gmcclins]

iptables -t nat -A POSTROUTING -s $dockernet -o ! docker0 -j MASQUERADE

Bad argument `docker0'

[theseer]

Try using SNAT:

iptables -t nat -A POSTROUTING -s $dockernet -o your_uplink_dev -j SNAT --to _your_uplink_ip

As long as you do not try to have publicly reachable services on docker, that should work to have a docker container reach the outside world.

[elnull]

@gmcclins Wrong arguments order. Should be iptables -t nat -A POSTROUTING -s $dockernet ! -o docker0 -j MASQUERADE

[gmcclins]

@elnull thank you.

[andreheuer]

@elnull I´m getting the error iptables v1.6.0: host/network !' not found` because the variable $dockernet is not defined. To what should I set the variable? BR

[elnull]

@andreheuer For example, if ip addr show dev docker0 | grep inet shows inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 then $dockernet should be 172.17.0.0/16.

[christophhaase]

I set the environment variable $dockernet as described. But it still doesn't work. The lxc container is not reachable outside the host. I also tried

sudo iptables -t nat -A POSTROUTING -s $dockernet ! -o docker0 -j SNAT --to ip_adress_of_container

because I use static ip adresses for the host and the container. But still no success. The only commands that makes the container reachable are

iptables -F FORWARD
iptables -P FORWARD ACCEPT

So what am I doing wrong?

[Cretection]

Is there a solution?

After every reboot, I have to set

iptables -F FORWARD
iptables -P FORWARD ACCEPT

again

[psycho0verload]

I've got a permanent solution using IPv4 Forwarding

[Wilhelmsson177]

I have the same problem as @Cretection the link provided by @Psycho0verload does not help, because IPv4 Forwarding is already enabled. Is there a good overall solution for fixing issues with LXC container and Docker?

[Cretection]

I solved it with edit the /etc/network/interfaces and add post-up iptables -F FORWARD && iptables -P FORWARD ACCEPT to the end. I found this later that my comment as @Psycho0verload not working.

[Cretection]

At last I got one more: sudo nano /etc/sysctl.conf uncomment: net.ipv4.ip_forward=1

[mattelacchiato]

All solutions described here did not work in case when restart=always is set for the docker container. I've compared the iptables in case of working and non-working, but they are identical but this line is present when it works: # Warning: iptables-legacy tables present, use iptables-legacy to see them

Although this output is the same for both cases (working & non-working):

$ sudo iptables-legacy -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

I've observed that the order is important: When starting the LXC container first and then the docker containers, it works. Other way around not.

My current workaround is to change /lib/systemd/system/docker.service and add an After dependency to the service which wraps my LXC container.

[kveroneau]

Hmm, by the sounds of this issue not being acknowledged by the docker developers, it seems that using Docker alongside another container technology such as LXC is not really supported. It would be nice if they made a small update to how Docker handles the FORWARD table to allow for new/existing rules to still work as expected. Mind you, I do have Docker running on a Linux router/gateway machine in my home, and I have some services set in the FORWARD table, and those custom iptables entries are unaffected. I wonder if perhaps the issue is related to how LXC is managing it's bridge interface, and uses the FORWARD table in an non-conventional way. I've been wanting to play around with LXC, so I may take a deeper look into what exactly is going on here, and causing LXC to lose it's connection. If I do manage to figure anything out at all, I'll be sure to post it into this issue for others who might try running both container technologies in unison. I do not see any reason why both container technologies could not run together on the same host.

I never ran into any issues on Debian 9 using a NAT behind a lxcbr0 bridge device. Here's what I did, I first stopped the docker daemon, cleared out everything docker placed into my iptables(shouldn't docker clear it's IP tables for me when it's stopped?), set the FORWARD to ACCEPT. Then, I proceeded to install lxc, lxcfs, and dnsmasq-base. bridge-utils was already installed. I configured lxc-net according to the documentation, and started that systemd service. This created my bridge interface, and added some rules into my iptables.

-A FORWARD -o lxcbr0 -j ACCEPT
-A FORWARD -i lxcbr0 -j ACCEPT

I then proceeded to create an lxc container, started it, and confirmed networking both into the container, and networking out to the world wide web was working as expected. Once that was all confirmed, I proceeded to start the docker daemon, where it then re-created it's own iptables rules. I then confirmed that both my docker and lxc containers had networking both inbound and outbound. Both container technologies appear to run on the same host together with no issues. I haven't tried a full host bridge network, where the LXC containers are fully exposed on the host's external network, but I am sure this will work even better as it shouldn't need to add any firewall rules, unless you need your LXC container to communicate back to your host over the network for some reason.

[the-real-ed]

I encountered the same issue, and was able to resolve it after reading the Docker documentation on how to setup Docker on a router.

In short, you need to add your custom rules to the DOCKER-USER table -- it appears that Docker respects these rules (i.e. doesn't clobber them).

For example, for my bridge interface br0, I added the following iptables rule:

-I DOCKER-USER -i br0 -o br0 -j ACCEPT

[mattelacchiato]

Thank you @the-real-ed ! That is really helpful! How do you make this rule persistent?

[the-real-ed]

@mattelacchiato in Debian-based systems, you can install the iptables-persistent package, and then add the rules to /etc/iptables/rules.v4.

The contents of the file should be something along the lines of:

*filter
-I DOCKER-USER -i br0 -o br0 -j ACCEPT
COMMIT

[nameless66]

Update System

sudo apt update && sudo apt upgrade

Install iptables-persistent

sudo apt-get install iptables-persistent # Save current rules for v4 & v6

Add Docker on a router iptables rule

sudo nano /etc/iptables/rules.v4 *filter :DOCKER-USER - [0:0] -I DOCKER-USER -i br0 -o br0 -j ACCEPT COMMIT

Bind lxc.network.link to br0

sudo sed -i '/lxc.network.link/c\lxc.network.link = br0' /etc/piVCCU3/lxc.config

[hoxu]

I ran into some networking issues while using lxc-net and installing docker, as well.

I'm not entirely sure what caused the networking issues, because after looking at the FORWARD chain rules, it should not matter whether lxc-net.service or docker.service is started first, either way ip filter FORWARD chain should end up in a working state.

$ sudo systemctl restart nftables lxc-net docker
$ sudo nft -s list chain ip filter FORWARD
table ip filter {
        chain FORWARD {
                type filter hook forward priority 0; policy drop;
                counter jump DOCKER-USER
                counter jump DOCKER-ISOLATION-STAGE-1
                oifname "docker0" ct state related,established counter accept
                oifname "docker0" counter jump DOCKER
                iifname "docker0" oifname != "docker0" counter accept
                iifname "docker0" oifname "docker0" counter accept
                oifname "lxcbr0" counter accept
                iifname "lxcbr0" counter accept
        }
}

Regardless, it may be a good idea to manually add lxc-net's FORWARD rules to DOCKER-USER chain as well.

Docker and iptables: Add iptables policies before Docker’s rules

If you need to add rules which load before Docker’s rules, add them to the DOCKER-USER chain. These rules are applied before any rules Docker creates automatically.

[mcx]

I solved it with edit the /etc/network/interfaces and add post-up iptables -F FORWARD && iptables -P FORWARD ACCEPT to the end.

At last I got one more: sudo nano /etc/sysctl.conf uncomment: net.ipv4.ip_forward=1

I can confirm that after performing both steps suggested by @Cretection networking is restored for my LXC containers, and Docker networking remains unaffected after a reboot.

[erik78se]

I got struck by this hard. I've got a thread here that shows the situation and also how I got networking back for lxd.

https://discuss.linuxcontainers.org/t/containers-fails-to-reach-internet/14671

[starkeepers]

The current solution to this problem in the LXD documentation doesn't work on Debian bookworm:

iptables -I DOCKER-USER -i <network_bridge> -o <external_interface> -j ACCEPT

I had to make it bidirectional for networking to start working again, in my case:

iptables -I DOCKER-USER -i lxdbr0 -o eno0 -j ACCEPT
iptables -I DOCKER-USER -i eno0 -o lxdbr0 -j ACCEPT

To make it stick across reboots, it can be added to /etc/network/interfaces as mentioned above, ie:

iface eno0 inet static
  ...
  post-up iptables -I DOCKER-USER -i lxdbr0 -o eno0 -j ACCEPT
  post-up iptables -I DOCKER-USER -i eno0 -o lxdbr0 -j ACCEPT

[simondeziel]

I had to make it bidirectional for networking to start working again, in my case:

FYI, we've updated the LXD documentation to have the returning traffic go through. This is checking the connection state rather than allowing all inbound traffic.