docker
Built-in DNS server extremely slow for large responses
- [x] I have tried with the latest version of my channel (Stable or Edge)
- [x] I have uploaded Diagnostics
- Diagnostics ID: 747C7FF1-4351-4543-B9E8-2B79CD4183A2/20200407164330
Expected behavior
When running DNS queries through docker's built-in DNS server, I expect similar performance (response times) compared to using external DNS servers directly.
Context
While the following reproduction case is an artificial example, it is a quite common case to have these huge DNS responses, especially when working with K8s ingress etc. I'm filing this issue because we essentially can't use Docker for Mac at our company because it breaks our DNS resolution (by timing out a lot).
Using an external DNS server directly is not an option for us since we also need DNS resolution for hostnames in the current docker network (for integration tests etc.)
Actual behavior
When executing DNS queries that yield large responses, it takes an extremely long time to return results. The results look correct though. This only happens on Docker for Mac. Running the same queries on linux-native docker does not show the same issue.
Information
- macOS Version: 10.15.4
- Docker Desktop Version: 2.2.0.5
Steps to reproduce the behavior
- To reproduce create a docker container with dig installed:
docker run -ti debian bash
apt-get update && apt-get install -y dnsutils
- Then run DNS queries that yield large responses in that container:
time dig hugedns.test.dziemba.net # ~5s
time dig hugedns.test.dziemba.net @ns-73-a.gandi.net # ~0.2s
- If you run queries that involve a lot of CNAMES, the performance gets even worse:
time dig cname5.hugedns.test.dziemba.net # ~18s
time dig cname5.hugedns.test.dziemba.net @ns-73-a.gandi.net # ~0.2s
As you can see, this only happens when docker's DNS server is used. When querying the external DNS server directly, performance is not impacted. When run on linux directly (not Docker for Mac), docker's built-in DNS server is also as fast as the external one.
Please let me know if you need more details.
Hey @djs55, maybe you can help out here? 🙂 You were already very helpful with a related-but-different problem a while back: https://github.com/docker/for-mac/issues/2160
Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.
Prevent issues from auto-closing with an /lifecycle frozen comment.
If this issue is safe to close now please do so.
Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale
@djs55 any idea where the bottleneck is? I can confirm that this is slow on Docker for Mac, but fast on a native Linux machine.
Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.
Prevent issues from auto-closing with an /lifecycle frozen comment.
If this issue is safe to close now please do so.
Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale
Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.
Prevent issues from auto-closing with an /lifecycle frozen comment.
If this issue is safe to close now please do so.
Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale
I've been suffering this issue for half a year.
Even for a normal size dns query, it is extremely slow, 30 seconds!
root@98f777f20ca9:/# cat /etc/resolv.conf
# DNS requests are forwarded to the host. DHCP DNS options are ignored.
nameserver 192.168.65.5
root@98f777f20ca9:/# time nslookup -v a.b.c
Server: 192.168.65.5
Address: 192.168.65.5#53
Non-authoritative answer:
Name: a.b.c
Address: 100.79.28.134
;; connection timed out; no servers could be reached
real 0m30.013s
user 0m0.005s
sys 0m0.008s
A ping command takes 15 seconds to resolve the dns name, see the strace result, it shows that the slow sys call happens in
sendto
recvfrom
The docker info is:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
scan: Docker Scan (Docker Inc., v0.5.0)
Server:
Containers: 147
Running: 5
Paused: 0
Stopped: 142
Images: 43
Server Version: 20.10.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.121-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 1.941GiB
Name: docker-desktop
ID: IGA3:HGMF:LFF4:QJIL:DKRS:FQ7D:OG6Y:W5AN:G7KE:QJQI:CRPH:XLO7
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 52
Goroutines: 65
System Time: 2021-03-19T06:31:49.480891297Z
EventsListeners: 4
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
This started happening for us in 2021 when we started migrating to m1 macs The issue is not present on intel macs
docker info
Client:
Version: 24.0.6
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2-desktop.5
Path: /Users/maxpatiiuk/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.22.0-desktop.2
Path: /Users/maxpatiiuk/.docker/cli-plugins/docker-compose
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: /Users/maxpatiiuk/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.20
Path: /Users/maxpatiiuk/.docker/cli-plugins/docker-extension
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v0.1.0-beta.8
Path: /Users/maxpatiiuk/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/maxpatiiuk/.docker/cli-plugins/docker-sbom
scan: Docker Scan (Docker Inc.)
Version: v0.26.0
Path: /Users/maxpatiiuk/.docker/cli-plugins/docker-scan
scout: Docker Scout (Docker Inc.)
Version: v1.0.7
Path: /Users/maxpatiiuk/.docker/cli-plugins/docker-scout
Server:
Containers: 9
Running: 8
Paused: 0
Stopped: 1
Images: 32
Server Version: 24.0.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
runc version: v1.1.8-0-g82f18fe
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.4.16-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 8
Total Memory: 7.667GiB
Name: docker-desktop
ID: 2653f9f0-5660-4b56-8c59-4e486c26a70b
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
WARNING: daemon is not using the default seccomp profile
more details in this issue: https://github.com/specify/specify7/issues/2574#issuecomment-2249202369
