Docker Network bypasses Firewall, no option to disable

[BenjamenMeyer]

Output of docker version:

Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 14
 Running: 5
 Paused: 0
 Stopped: 9
Images: 152
Server Version: 1.10.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 204
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.13.0-58-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.793 GiB
Name: brm-pheonix-dev
ID: Y6Z4:6D53:RFOL:Z3CM:P7ZK:H6HL:RLV5:JT73:LZMC:DTBD:7ILK:2RS5
Username: benjamenmeyer
Registry: https://index.docker.io/v1/

Additional environment details (AWS, VirtualBox, physical, etc.): Rackspace Cloud Server, Ubuntu 14.04, but that shouldn't really matter

Steps to reproduce the issue:

1. Setup the system with a locked down firewall
2. Create a set of docker containers with exposed ports
3. Check the firewall; docker will by use "anywhere" as the source, thereby all containers are exposed to the public.

Describe the results you received: root@brm-pheonix-dev:~/rse# iptables --list DOCKER Chain DOCKER (1 references) target prot opt source destination
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:6379

Describe the results you expected: root@brm-pheonix-dev:~/rse# iptables --list DOCKER Chain DOCKER (1 references) target prot opt source destination
ACCEPT tcp -- 127.0.0.0/24 172.17.0.2 tcp dpt:6379 ACCEPT tcp -- 172.16.0.0/16 172.17.0.2 tcp dpt:6379

Additional information you deem important (e.g. issue happens only occasionally):

By default docker is munging the firewall in a way that breaks security - it allows all traffic from all network devices to access the exposed ports on containers. Consider a site that has 2 containers: Container A exposes 443 running Nginx, and Container B runs an API on port 8000. It's desirable to open Container A to the public for use, but hide Container B entirely so that it can only talk to localhost (for testing by the user) and the docker network (for talking to Container A). It might also be desirable for testing purposes to have Container C be a database used by Container B with the same kind of restrictions.

I found this because of monitoring logs on a service I had thought was not open to the public. After finding log entries from sources trying to break in, I checked the firewall rules and found there was no limit on the source addresses or interfaces. I use UFW and only allow SSH onto this particular box, and would prefer to keep it that way. This can dramatically impact using Docker containers to deploy services and lead to potential security problems if people are not careful.

The best security practice would be to by default limit the networking to work like above desired effect example, and then allow the user to add the appropriate firewall, etc rules to override such behavior, or have an option to revert to the current behavior. I know that for legacy reasons that is not likely since it would break a lot of things on up-date; so at least having an option to enable the above that can be turned on now would be a good first step, and perhaps later after much warning make it the default behavior. Assuming the default behavior is secure, having functionality to manage this (firewall->enable public port, ip) in the docker-compose yml would be a great way to visibly make it known what is going on.

I did find the --iptables=false option, however, I don't want to have to be setting all the rules myself. The only thing I am objecting to is the source setting for the rules.

While I have not verified it, I suspect all the firewalls supported by docker will have the same issue.

[BenjamenMeyer]

Note: I noticed in https://github.com/docker/docker/blob/master/vendor/src/github.com/docker/libnetwork/iptables/iptables.go that there is not even an existing option to set the source, so it's just using the iptables defaults for source ip/device.

[thaJeztah]

Duplicate of https://github.com/docker/docker/issues/14041?

[justincormack]

It is not quite #14041 as this issue is talking about exposed ports. Exposing ports is intended to make them publicly accessible, as this is how you expose services to the outside world. If you are working in a development environment, you can either disable access to ports from outside your computer with a host firewall, or simply not expose the ports and access the services directly, or from other containers on the same network.

I would recommend you use the newer docker networking features to set up private networks for services that you do not want exposed at all, see https://docs.docker.com/engine/userguide/networking/

[thaJeztah]

That's what I thought of first; but was a bit confused, because exposing a port (EXPOSE), doesn't actually do anything, but publishing a port (-p / -P) actually exposes it on the host.

If you're actually talking about publishing, then this is as designed;

In your example, container B and C should not publish their ports, and container A can communicate with them through the Docker Network, e,g.

docker network create mynet

docker run -d --net=mynet --name=api api-image
docker run -d --net=mynet --name=db database-image
docker run -d --net=mynet --name=web -p 443:443 nginx

This only publishes the "web" container to the host, The web container can access the "API" and "database" containers through their name (I.e. http://api:80/, and db:3306 (assuming MySQL))

[BenjamenMeyer]

@justincormack so I don't think using a private network solves the issue. In my case I'm using a private network between the containers, and they're still publicly exposed because the host firewall isn't configured to limit the exposure to the private network.

@thaJeztah the issue still comes down to the firewall support - there's no firewall support in docker to limit it to a specific network. You can probably still access those containers from another system as the firewall will not prevent other systems from accessing the port on the host.

Now I'm running this via docker-compose; however, it's not entirely a docker-compose issue since the libnetwork functionality has no capability of limiting the network in the firewall rules - iptables rules have no source specification so regardless of how one configures the network as long as one relies on docker to create the firewall rules (which one should because it's more likely to get them right) then this becomes an issue. Consider the following in a docker-compose.yml file:

nginx:
    build: ./docker/nginx/.
    ports:
        - "127.0.0.1:8080:80"
        - "127.0.0.1:443:443"
    environment:
        DESTINATION_HOST: repose
    links:
        - repose
repose:
    build: ./docker/repose/.
    ports:
        - "127.0.0.1:80:8080"
    environment:
        DESTINATION_HOST: phoenix
        DESTINATION_PORT: 8888
    links:
        - phoenix
curryproxy:
    build: ./docker/curryproxy/.
    ports:
        - "127.0.0.1:8081:8081"
    external_links:
        - rse_rse_1
        - rse_rse_2
        - rse_rse_3
phoenix:
    build: .
    ports:
        - '127.0.0.1:88:8888'
    links:
        - curryproxy:curry
    external_links:
        - rse_rse_1:rse
        - rse_rse_2
        - rse_rse_3
        - rse_cache_1:cache
    volumes:
        - .:/home/phoenix

The above is an excerpt from one of my projects. While I want to be able to test all of them locally from my host, I don't want anyone else to be able to access anything but the nginx instance.

I'm not sure how this translates to your nomenclature...it may be that this is part of the "publishing" aspect, and the publishing capability needs to be expanded to do what I'm saying.

If this is by design, then it's a poor security model as you now expose all developers to extreme risks when on unfamiliar networks (e.g traveling).

As I said, I don't expect the default to change immediately but having the option would be a good first step.

[justincormack]

I am a bit confused then, can you give some examples externally of what you can connect to? The backend services will be (by default) on the 172.17.0.0/16 network, which you will not be able to access externally I wouldn't think first because you will not have a route to that defined from an external host.

There is a potential issue if your external IP is also a private IP that traffic will not be dropped that is routed for the internal networks (whereas it should be from public to private) - is that the issue?

[BenjamenMeyer]

@justincormack so I'm primarily setting up proper proxying so that some services can only be hit via the proxy (nginx - ssl termination), which then filters through an authentication proxy (repose), and finally off to another service (phoenix). I could care less if all of them are bound to the 0.0.0.0 interface; but I only want nginx to be externally accessible (or at the very least the repose portion if I didn't have nginx in place here). An easy solution, for example, would be to not have to set "127.0.0.1" in the configuration, but have a firewall section where it's easy to specify that to allow through the firewall with a base configuration of only the docker network and local host (loopback) interfaces enabled to talk - something like:

firewall:
    external:
        ports:
            - 80
            - 443

Now the situation can be mitigated somewhat by limited the network mapping on the host to 127.0.0.1 instead of the default 0.0.0.0 map. Note that this is what really mitigates it because otherwise the bridging will forward the host port into the docker network.

And yes, I did verify that that limiting works; however, it still leaves potential vulnerabilities in place and the firewall rules do not match what is actually being done.

As another example, there was a Linux Kernel vulnerability a little while back (having trouble finding it at the moment) that was related to ports that were marked in IPtables as being opened for use by applications, but then not actually being connected to an application - for instance, being on a local host port but not a public IP port. This potentially sets that up, and it would be better practice to limit the IPtables rules to the expected networks instead of leaving them open to connect from any where. As I said, at the very least have the option to specify. They've likely fixed that particular issue but why leave the possibility open?

IOW, it's all about security.

[thaJeztah]

@BenjamenMeyer if you don't want the other services to be accessible, why do you publish their ports at all? i.e. "127.0.0.1:8081:8081" is not needed if it's only accessed through the docker network (other services connect directly through the docker network)

[seeruk]

One issue I have that's related to this one is that I would like to publish ports, but only allow certain IP addresses to access them.

For example, I run a Jenkins environment in a couple of containers. The master node is "published", but I have to make some pretty convoluted iptables rules to lock it down so that only the 2 offices we have can access it.

Is there a way around this currently built into Docker? Or at least a recommended practice? I've seen in the documentation how you might restrict access to 1 IP address; but not several. The other issue with this is that if you have a server that already has an iptables configuration, you might be resetting all of the rules before applying your rules (hence the convoluted rules I have had to set up).

[enzeart]

I have an issue similar to the one stated by @SeerUK. There is a jarring violation of expectation when preexisting firewall rules don't apply to the published container ports. The desired behavior is as follows (for me at least)

1. User connection attempt is filtered based on INPUT configurations, etc
2. Traffic forwarding then happens as usual based on the docker-added FORWARD rules

Is there a succinct way to achieve this in iptables, or does it not easily permit such a construct. I'm particularly limited in my knowledge of iptables so bear with me. I've just recently picked up knowledge about it while trying to understand docker's interactions with it.

[seeruk]

What I've actually resorted to for the time being, since I am actually running these containers on a pretty powerful dedicated server, I've set up a KVM VM running Docker, then using some more standard iptables rules to restrict access from the host. The VM has it's own network interface that's only accessibly from the server, so I have to add rules to explicitly allow access to ports in iptables on the host. I have lost a little bit of performance, but not much.

[BenjamenMeyer]

@thaJeztah I want to be able to access it from the local system, and test against it easily. For example, setting up a RESTful HTTP API that has a Health end-point and being able to reliably run curl against it by using localhost (I have to document this for others and having IP addresses that change is not reliable). In most cases for my dev environment I only want the containers to talk to each other, but I also want to be able to access it from the host.

For @SeerUK's case, being able set an IP block (5.5.0.0/16 - a valid parameter for a source address in iptables rules) would be a very good thing. IPtables already has the capability to do the limiting, but docker is not taking advantage of it.

[BenjamenMeyer]

@thaJeztah I set "127.0.0.1:8081:8081" explicitly to keep it off the external network; I had found logs in my docker containers from people trying to crack into the containers via the exposed ports.

My work around right now is to turn off docker containers before I leave for the day because I can't ensure the environment I want to be external actually is external, or that the environment is properly limited for security purposes.

[thaJeztah]

@BenjamenMeyer one way to do this is running those tests in a container, e.g.

docker run --net -it --rm --net=mynetwork healthchecker 

[jcheroske]

The issue that Ben is bringing to light is real and surprising (a bad combination). Many admins, like myself, are using the tried-and-true ufw firewall. Docker is doing and end-run around ufw and altering the iptables rules is in such a way that it 1) causes ufw to misreport the current status of the packet filtering rules, and 2) exposes seemingly private services to the public network. In order for docker to remain in the good graces of the sysadmin community, another approach must be devised. Right now there are many admins out there, who, like Ben and myself, inadvertently opened ports to the wider Internet. Unlike Ben and myself though, they have not figured it out yet.

[BenjamenMeyer]

@thaJeztah that assumes that I am doing it via the command-line and not using another tool where I only have to set an IP address.

For example, I'm working on an API. I have a tool that I can use to work with that API in production to support it; for development of the tool and the API I want to just point the tool at the dockerized API. The tool knows nothing about docker, nor should it. And I don't necessarily want to put the tool into docker just to use it - pointing it at a port exposed only to the local host should be sufficient.

[BenjamenMeyer]

@jcheroske I agree, but I don't know that there's a good solution to that aspect. For that, ufw probably needs to be made smarter to be able to lookup and report on rules that it wasn't involved in creating. There's lot of software out there that can adjust the iptables rules in ways that ufw (or AFAIK firewalld, etc) won't know about. There's not really a simple solution to fixing that either.

That said, it would be nice if Docker could integrate with those to dump out the appropriate config files to be able to enable/disable them, or integrate with those tools such that it gets hooked in and dumps out the information appropriately, however, given there are better solutions I don't think that aspect will really be solved. Here, it's more about just limiting the scope of the iptables rules that are being generated to at least minimize the potential impacts by allowing the specification of the source (lo, eth0, 127.0.0.0/24, etc).

[seeruk]

If you are willing to do so, using iptables does make this totally possible.

This is a trimmed-down example of how you can use it: https://gist.github.com/SeerUK/b583cc6f048270e0ddc0105e4b36e480

You can see that right at the bottom, 1.2.3.4 is explicitly given access to port 8000 (which is exposed by Docker), then anything else to that port is dropped. The PRE_DOCKER chain is inserted to be before the DOCKER chain so that it is hit first, meaning the DROP stops the blocked requests from ever reaching the DOCKER chain.

It's a bit annoying that Docker doesn't have this functionality built-in, but it is possible work around it right now.

Another alternative would be using an external firewall. Some places like AWS and Scaleway offer things like security groups where you can manage access to your boxes from outside, from there every port behaves the same way.

I never actually managed to figure out how to make this work with UFW. Though for now, I'm happy with using iptables as a solution. It seems to be working out very well for me so far.

Obviously, this isn't much of a great solution if you have already built a reasonably complicated set of firewall rules around UFW. Though, it does make using things like iptables-persistent quite easy. You can also use alternative ways of allowing access to this way that seem more "normal" in iptables.

[mavenugo]

@BenjamenMeyer have you thought of using an user-defined docker network with a subnet & ip-range option and assigning a static ip-address for containers & using them for local development so that you don't have to depend on a virtual static ip such as 127.0.0.1 ? That will avoid the need to have port-mapping all together for those containers that are private to a host.

docker network create --subnet=30.1.0.0/16 --ip-range=30.1.0.0/24 mynetwork docker run --net=mynetwork --ip=30.1.1.1 --name=myservice1 xxxx docker run --net=mynetwork --ip=30.1.1.2 --name=myservice2 yyyy

With this setup, myservice2 can reach myservice1 by name myservice1 and there is no need to even depend on the static ip. Also the host can reach the static-ip freely without the need to have port-mapping.

Also with compose 1.7, you can specify static ip address for containers and specify network subnets and ranges.

[jcheroske]

I did figure out a simple workaround.

1. Edit /etc/default/docker: DOCKER_OPTS="--iptables=false"

2. Add ufw rule: ufw allow to <private_ip> port <port>

So simple that it really makes me wonder why the --iptables=false option is not the default. Why create such a situation when all docker has to do is say, "Hey, if you're running a firewall you're going to have to punch a hole through it!" What am I missing?

https://fralef.me/docker-and-iptables.html http://blog.viktorpetersson.com/post/101707677489/the-dangers-of-ufw-docker

[enzeart]

I can't get docker to stop modifying iptables to save my life. Tried updating /etc/default/docker to no avail on Ubuntu 16.04

[seeruk]

@enzeart Try /lib/systemd/system/docker.service.

[enzeart]

@SeerUK Bless your soul

[thaJeztah]

@enzeart to configure a daemon running on a host that uses systemd, it's best to not edit the docker.unit file itself, but to use a "drop in" file. That way, you won't run into issues when upgrading docker (in case there's a newer docker.unit file). See https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options for more info.

You can also use a daemon.json configuration file, see https://docs.docker.com/engine/reference/commandline/daemon/#daemon-configuration-file

[BenjamenMeyer]

@mavenugo There's already a docker network in place.

@jcheroske that works, but as I noted it would mean that the end-user (me) would then have to make sure that all iptables rules were correct, which is not optimal and not nearly as likely to happen as having docker do it automatically, thus this issue.

[westsouthnight]

Hi, please up. I think its issue too. Containers chain in Iptables need be after main rules and not be exposed to the world by default.

[StefanPanait]

I would really like to see Docker (and docker-compose) having the ability to whitelist or blacklist IPs that can access that port.

Take for example:

nginx:
    ports:
      - "8000:8000"
    whitelist:
      - 10.6.20.2

Would imply that only a source IP of 10.6.20.2 could access port 8000 on this host.

[seeruk]

@StefanPanait I really like that idea. It could also work with a similar syntax to volumes and access/deny lists, something like:

nginx:
  access:
  - "10.0.1.6:allow"
  - "deny"

It would of course still have to allow things like inter-container communication.

[BenjamenMeyer]

@SeerUK inter-container communication should be a default, but why should you not be able to prohibit one container from talking to it? That could be extremely useful for debugging...

Though I guess the proper way to do that would be to separate the docker networks...still, I think being able to do something like:

nginx: access: - "10.0.1.6:allow" - "webapi:allow" - "database:deny" - "deny" Could be useful...question is, is it useful enough to justify implementation to that degree? I don't know.

At present, I'd like to see the original issue resolved, then features like this can be added if they don't make sense to design into the resolution to start with (they might).

[thaJeztah]

why should you not be able to prohibit one container from talking to it? That could be extremely useful for debugging

That's were docker network disconnect is for? You can disconnect a container from the network for debugging, and reattach it with docker network attach