docker
dockerd should use 127.0.0.53 when requested
- [x] This is a bug report
- [ ] This is a feature request
- [ ] I searched existing issues before opening this one
Expected behavior
When specifying {"dns":["127.0.0.53"]} in daemon.json, dockerd would use that as the DNS server.
Actual behavior
When you do this, dockerd helpfully notes that it's ignoring your request:
INFO[2020-04-15T19:47:48.994402261-07:00] detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf
Unfortunately, this is not that same... and in fact it's much worse. For example, if you VPN, resolved will intelligently route the right DNS request to the right DNS servers on the right interfaces based on their domain. That way you can just query against 127.0.0.53 and things work.
However, the back-compat /run/systemd/resolve/resolv.conf has a list of all DNS servers on all interfaces from all services. This means that if have more than 1 "home" DNS server, then the DNS servers handed to you by your VPN client will be further down the list and dockerd will never query them.
If dockerd would actually query the requested address it would avoid bugs like this one and this one.
Steps to reproduce the behavior
- set dns to
127.0.0.53 - watch it ignore that
Output of docker version:
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:25:46 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:24:19 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Output of docker info:
Client:
Debug Mode: false
Server:
Containers: 18
Running: 0
Paused: 0
Stopped: 18
Images: 1
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.0.0-31-generic
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.32GiB
Name: ldx-vulturus.corp.vicarious.com
ID: 5KZR:CPJV:KHZ4:MGWP:GR3G:NMUM:62QX:OJCE:XTSZ:IQ6Z:5UEY:JGA3
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.) Ubuntu 18.04
The issue is it's not really possible to use 127.0.0.53 from the host in the container.
This is dockerds config, right? Why cannot the daemon talk to things on the host?
Also, what about --network host containers?
@jaymzh thanks a lot for submitting this issue. I agree that semantics of using systemd resolver is not same to just using /run/systemd/resolve/resolv.conf as /etc/resolv.conf.
@cpuguy83 can't Docker make its embedded DNS resolver route DNS queries to systemd-resolver by default? Or provide another embedded DNS resolver with the functionality mentioned which can be enabled on demand for such a case? Or option to always use embedded DNS resolver with another option to enable mentioned functionality?
I opened a pull request that may partially address this issue; https://github.com/moby/moby/pull/41022. However, using 127..x.x resolvers will (as mentioned) only work if a custom network is used, as the embedded DNS is disabled when using the default bridge network.
not having the embedded DNS available on the default bridge network was done for backward compatibility with older releases; I did open a proposal to deprecate that altogether and always have the embedded DNS available (https://github.com/moby/moby/issues/22295), but no decision was made on that yet.
FWIW, the situation on Docker for Mac (https://github.com/docker/for-mac/issues/2820) may be more involved, because there's more network "magic" involved to integrate the docker daemon (running in a lightweight VM) with the host's network stack; best to discuss that on the Docker for Mac issue
The workaround linked by cohoe above (https://github.com/cohoe/workstation/issues/105) solved the issue for me (i.e. have systemd-resolved listen on a secondary IP, and point docker to that). In my case, I'm using systemd-resolved to ensure certain domains are looked up using VPN'ed DNS servers
The workaround linked by cohoe above (cohoe/workstation#105) solved the issue for me (i.e. have systemd-resolved listen on a secondary IP, and point docker to that). In my case, I'm using systemd-resolved to ensure certain domains are looked up using VPN'ed DNS servers
In case someone runs into this, here is the solution we went for: The workaround indeed works, but it has to be made sure that the secondary IP is reachable from inside the docker containers (duh?). Since containers are in a network namespace, IPs in local ranges on the host, such as 127.0.0.54 or the mentioned 172.17.0.1, would not work for us (you can verify this with nsenter + ping). Instead, we have made systemd-resolved expose its stub DNS server to an IP address attached to a physical network interface of the machine, which is then reachable from inside the containers, akin to how public internet IP addresses are reachable. Note that depending on your setup running a local DNS server on a physical network interface of your machine might be a security issue unless further firewalling is implemented.
