dns icon indicating copy to clipboard operation
dns copied to clipboard

Published 20 hours ago • verified publisher icon qdm12

Plaintext DNS queries leaking despite UPSTREAM_TYPE=DoT

Open elduds opened this issue 10 months ago • 2 comments

Howdy, I am witnessing plaintext DNS queries being sent out despite setting UPSTREAM_TYPE=DoT. The overwhelming majority of queries are via DoT but quite a few are being sent plaintext.

Just wondering if I've got some configuration wrong. I can't see any option to disable plaintext as the PLAIN_RESOLVERS environment variable does not have an option to disable, however I would assume that with UPSTREAM_TYPE=DoT that PLAIN_RESOLVERS is not applicable.

Expected behaviour: When setting UPSTREAM_TYPE=DoT, only DoT queries should ever be sent to upstreams. Plaintext queries should never be sent under any conditions.

Observed behaviour: Sometimes plaintext queries are sent. See packet captures and logs

03:09:41.480317 IP 172.17.0.1.51485 > 172.17.0.2.53: 61232+ [1au] A? tRaCKEr.ToRREnt.Eu.oRG. (63)
03:09:41.480876 IP 172.17.0.2.37505 > 8.8.8.8.53: 61232+ [1au] A? tRaCKEr.ToRREnt.Eu.oRG. (63)

In the below packet captures, 172.17.0.1 - client 172.17.0.2 and 2400:a840:405b:13a0:0:242:ac11:2 - dns server container (this project)

docker-compose file:

version: "3.7"
services:
  dns:
    image: qmcgaw/dns:v2.0.0-beta
    container_name: dns
    environment:
      - LOG_LEVEL=debug
      - UPSTREAM_TYPE=DoT
      - DOT_RESOLVERS=cloudflare,google
      - CACHE_TYPE=noop
      - CHECK_DNS=off
      - BLOCK_MALICIOUS=off
      - BLOCK_SURVEILLANCE=off
      - BLOCK_ADS=off
    network_mode: bridge
    restart: always

========================================
========================================
================= dns ==================
========================================
=========== Made with ❤️ by ============
======= https://github.com/qdm12 =======
========================================
========================================

Running version v2.0.0-beta built on 2024-11-04T16:43:20.157Z (commit 111c6a1)

🔧 Need help? ☕ Discussion? https://github.com/qdm12/dns/discussions/new/choose
🐛 Bug? ✨ New feature? https://github.com/qdm12/dns/issues/new/choose
💻 Email? [email protected]
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2025-01-29T03:04:40Z INFO Settings:
├── DNS upstream connection: dot
├── DNS server listening address: :53
├── DNS over TLS:
|   ├── Upstream resolvers: cloudflare and google
|   └── Request timeout: 3s
├── Cache:
|   └── Type: noop
├── Filtering:
|   └── Blocked categories:
├── Logging:
|   ├── Level: debug
|   └── Caller: hidden
├── Middleware logging: disabled
├── Metrics:
|   └── Type: noop
├── Local DNS middleware:
|   └── Local resolvers:
|       ├── 8.8.8.8:53
|       ├── 8.8.4.4:53
|       ├── [2001:4860:4860::8888]:53
|       └── [2001:4860:4860::8844]:53
├── Substitute middleware: disabled
├── Check DNS: no
└── Periodic update: every 24h0m0s
2025-01-29T03:04:40Z INFO using DNS address 127.0.0.1 internally
2025-01-29T03:04:40Z DEBUG [services] dns loop starting
2025-01-29T03:04:40Z INFO [DNS server loop] IPv6 is supported, communicating with upstream resolvers only over IPv6
2025-01-29T03:04:40Z INFO [DNS server loop] starting DNS server
2025-01-29T03:04:40Z INFO [DNS over TLS] DNS server listening on [::]:53
2025-01-29T03:04:40Z INFO [DNS server loop] downloading and building DNS block lists
2025-01-29T03:04:40Z INFO [DNS server loop] 0 hostnames blocked overall
2025-01-29T03:04:40Z INFO [DNS server loop] 0 IP addresses blocked overall
2025-01-29T03:04:40Z INFO [DNS server loop] 0 IP networks blocked overall
2025-01-29T03:04:40Z INFO [DNS server loop] starting DNS server
2025-01-29T03:04:40Z INFO [DNS over TLS] DNS server listening on [::]:53
2025-01-29T03:04:40Z DEBUG [services] dns loop started
2025-01-29T03:04:40Z DEBUG [services] dummy metrics server starting
2025-01-29T03:04:40Z DEBUG [services] dummy metrics server started
2025-01-29T03:04:40Z DEBUG [services] health http server starting
2025-01-29T03:04:40Z INFO [health server] health http server listening on 127.0.0.1:9999
2025-01-29T03:04:40Z DEBUG [services] health http server started

Example of plaintext leakage:

root@gl-mt6000:~# dig @172.17.0.2 tRaCKEr.ToRREnt.Eu.oRG

; <<>> DiG 9.18.28 <<>> @172.17.0.2 tRaCKEr.ToRREnt.Eu.oRG
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61232
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;tRaCKEr.ToRREnt.Eu.oRG.                IN      A

;; ANSWER SECTION:
tRaCKEr.ToRREnt.Eu.oRG. 15      IN      A       91.216.110.53

;; Query time: 40 msec
;; SERVER: 172.17.0.2#53(172.17.0.2) (UDP)
;; WHEN: Wed Jan 29 03:09:41 UTC 2025
;; MSG SIZE  rcvd: 89

root@gl-mt6000:/mnt/infra/wrt/dns# tcpdump -ni docker0 port 53
03:09:41.480317 IP 172.17.0.1.51485 > 172.17.0.2.53: 61232+ [1au] A? tRaCKEr.ToRREnt.Eu.oRG. (63)
03:09:41.480876 IP 172.17.0.2.37505 > 8.8.8.8.53: 61232+ [1au] A? tRaCKEr.ToRREnt.Eu.oRG. (63)
03:09:41.521499 IP 8.8.8.8.53 > 172.17.0.2.37505: 61232 1/0/1 A 91.216.110.53 (67)
03:09:41.521751 IP 172.17.0.2.53 > 172.17.0.1.51485: 61232 1/0/1 A 91.216.110.53 (89)

Example of correct working DoT query

root@gl-mt6000:/mnt/infra/wrt/dns# tcpdump -ni docker0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on docker0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
03:18:56.710264 IP 172.17.0.1.40482 > 172.17.0.2.53: 37995+ [1au] A? www.abc.net.au. (55)
03:18:56.710788 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [S], seq 1840091009, win 64800, options [mss 1440,sackOK,TS val 338328234 ecr 0,nop,wscale 7], length 0
03:18:56.749892 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [S.], seq 1888515637, ack 1840091010, win 65535, options [mss 1440,sackOK,TS val 2527296082 ecr 338328234,nop,wscale 8], length 0
03:18:56.749958 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 1, win 507, options [nop,nop,TS val 338328273 ecr 2527296082], length 0
03:18:56.751157 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [P.], seq 1:1473, ack 1, win 507, options [nop,nop,TS val 338328274 ecr 2527296082], length 1472
03:18:56.790872 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], ack 1, win 1055, options [nop,nop,TS val 2527296123 ecr 338328273,nop,nop,sack 1 {1429:1473}], length 0
03:18:56.790893 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], ack 1473, win 1050, options [nop,nop,TS val 2527296123 ecr 338328274], length 0
03:18:56.873655 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], seq 1:1209, ack 1473, win 1050, options [nop,nop,TS val 2527296205 ecr 338328274], length 1208
03:18:56.873705 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 1209, win 501, options [nop,nop,TS val 338328397 ecr 2527296205], length 0
03:18:56.873763 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [P.], seq 1209:2417, ack 1473, win 1050, options [nop,nop,TS val 2527296205 ecr 338328274], length 1208
03:18:56.873786 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 2417, win 499, options [nop,nop,TS val 338328397 ecr 2527296205], length 0
03:18:56.873877 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], seq 2417:3625, ack 1473, win 1050, options [nop,nop,TS val 2527296205 ecr 338328274], length 1208
03:18:56.873899 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 3625, win 490, options [nop,nop,TS val 338328397 ecr 2527296205], length 0
03:18:56.873938 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [P.], seq 3625:4833, ack 1473, win 1050, options [nop,nop,TS val 2527296205 ecr 338328274], length 1208
03:18:56.873951 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [P.], seq 4833:5326, ack 1473, win 1050, options [nop,nop,TS val 2527296205 ecr 338328274], length 493
03:18:56.873964 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 4833, win 485, options [nop,nop,TS val 338328397 ecr 2527296205], length 0
03:18:56.873969 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 5326, win 482, options [nop,nop,TS val 338328397 ecr 2527296205], length 0
03:18:56.880573 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [P.], seq 1473:1537, ack 5326, win 501, options [nop,nop,TS val 338328403 ecr 2527296205], length 64
03:18:56.880803 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [P.], seq 1537:1616, ack 5326, win 501, options [nop,nop,TS val 338328404 ecr 2527296205], length 79
03:18:56.919850 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], ack 1537, win 1050, options [nop,nop,TS val 2527296252 ecr 338328403], length 0
03:18:56.919867 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], ack 1616, win 1050, options [nop,nop,TS val 2527296252 ecr 338328404], length 0
03:18:56.923432 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [P.], seq 5326:5482, ack 1616, win 1050, options [nop,nop,TS val 2527296255 ecr 338328404], length 156
03:18:56.923610 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [P.], seq 1616:1640, ack 5482, win 501, options [nop,nop,TS val 338328447 ecr 2527296255], length 24
03:18:56.923703 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [F.], seq 1640, ack 5482, win 501, options [nop,nop,TS val 338328447 ecr 2527296255], length 0
03:18:56.923860 IP 172.17.0.2.53 > 172.17.0.1.40482: 37995 3/0/1 CNAME www.abc.net.au.edgekey.net., CNAME e3161.b.akamaiedge.net., A 23.208.184.139 (197)
03:18:56.963604 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [F.], seq 5482, ack 1641, win 1050, options [nop,nop,TS val 2527296295 ecr 338328447], length 0
03:18:56.963667 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 5483, win 501, options [nop,nop,TS val 338328487 ecr 2527296295], length 0


root@gl-mt6000:~# dig www.abc.net.au @172.17.0.2

; <<>> DiG 9.18.28 <<>> www.abc.net.au @172.17.0.2
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37995
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.abc.net.au.                        IN      A

;; ANSWER SECTION:
www.abc.net.au.         258     IN      CNAME   www.abc.net.au.edgekey.net.
www.abc.net.au.edgekey.net. 9305 IN     CNAME   e3161.b.akamaiedge.net.
e3161.b.akamaiedge.net. 20      IN      A       23.208.184.139

;; Query time: 210 msec
;; SERVER: 172.17.0.2#53(172.17.0.2) (UDP)
;; WHEN: Wed Jan 29 03:18:56 UTC 2025
;; MSG SIZE  rcvd: 197
root@gl-mt6000:/mnt/infra/wrt/dns# tcpdump -ni docker0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on docker0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
03:18:56.710264 IP 172.17.0.1.40482 > 172.17.0.2.53: 37995+ [1au] A? www.abc.net.au. (55)
03:18:56.710788 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [S], seq 1840091009, win 64800, options [mss 1440,sackOK,TS val 338328234 ecr 0,nop,wscale 7], length 0
03:18:56.749892 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [S.], seq 1888515637, ack 1840091010, win 65535, options [mss 1440,sackOK,TS val 2527296082 ecr 338328234,nop,wscale 8], length 0
03:18:56.749958 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 1, win 507, options [nop,nop,TS val 338328273 ecr 2527296082], length 0
03:18:56.751157 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [P.], seq 1:1473, ack 1, win 507, options [nop,nop,TS val 338328274 ecr 2527296082], length 1472
03:18:56.790872 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], ack 1, win 1055, options [nop,nop,TS val 2527296123 ecr 338328273,nop,nop,sack 1 {1429:1473}], length 0
03:18:56.790893 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], ack 1473, win 1050, options [nop,nop,TS val 2527296123 ecr 338328274], length 0
03:18:56.873655 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], seq 1:1209, ack 1473, win 1050, options [nop,nop,TS val 2527296205 ecr 338328274], length 1208
03:18:56.873705 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 1209, win 501, options [nop,nop,TS val 338328397 ecr 2527296205], length 0
03:18:56.873763 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [P.], seq 1209:2417, ack 1473, win 1050, options [nop,nop,TS val 2527296205 ecr 338328274], length 1208
03:18:56.873786 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 2417, win 499, options [nop,nop,TS val 338328397 ecr 2527296205], length 0
03:18:56.873877 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], seq 2417:3625, ack 1473, win 1050, options [nop,nop,TS val 2527296205 ecr 338328274], length 1208
03:18:56.873899 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 3625, win 490, options [nop,nop,TS val 338328397 ecr 2527296205], length 0
03:18:56.873938 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [P.], seq 3625:4833, ack 1473, win 1050, options [nop,nop,TS val 2527296205 ecr 338328274], length 1208
03:18:56.873951 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [P.], seq 4833:5326, ack 1473, win 1050, options [nop,nop,TS val 2527296205 ecr 338328274], length 493
03:18:56.873964 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 4833, win 485, options [nop,nop,TS val 338328397 ecr 2527296205], length 0
03:18:56.873969 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 5326, win 482, options [nop,nop,TS val 338328397 ecr 2527296205], length 0
03:18:56.880573 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [P.], seq 1473:1537, ack 5326, win 501, options [nop,nop,TS val 338328403 ecr 2527296205], length 64
03:18:56.880803 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [P.], seq 1537:1616, ack 5326, win 501, options [nop,nop,TS val 338328404 ecr 2527296205], length 79
03:18:56.919850 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], ack 1537, win 1050, options [nop,nop,TS val 2527296252 ecr 338328403], length 0
03:18:56.919867 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [.], ack 1616, win 1050, options [nop,nop,TS val 2527296252 ecr 338328404], length 0
03:18:56.923432 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [P.], seq 5326:5482, ack 1616, win 1050, options [nop,nop,TS val 2527296255 ecr 338328404], length 156
03:18:56.923610 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [P.], seq 1616:1640, ack 5482, win 501, options [nop,nop,TS val 338328447 ecr 2527296255], length 24
03:18:56.923703 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [F.], seq 1640, ack 5482, win 501, options [nop,nop,TS val 338328447 ecr 2527296255], length 0
03:18:56.923860 IP 172.17.0.2.53 > 172.17.0.1.40482: 37995 3/0/1 CNAME www.abc.net.au.edgekey.net., CNAME e3161.b.akamaiedge.net., A 23.208.184.139 (197)
03:18:56.963604 IP6 2001:4860:4860::8888.853 > 2400:a840:405b:13a0:0:242:ac11:2.59806: Flags [F.], seq 5482, ack 1641, win 1050, options [nop,nop,TS val 2527296295 ecr 338328447], length 0
03:18:56.963667 IP6 2400:a840:405b:13a0:0:242:ac11:2.59806 > 2001:4860:4860::8888.853: Flags [.], ack 5483, win 501, options [nop,nop,TS val 338328487 ecr 2527296295], length 0

Cheers!

elduds avatar Jan 29 '25 03:01 elduds

I'll do some more testing to try to constrain the conditions under which this occurs, and maybe take a look at the code to see what might be happening

elduds avatar Jan 29 '25 03:01 elduds 

REPOSITORY   TAG           IMAGE ID       CREATED        SIZE
qmcgaw/dns   v2.0.0-beta   0d7752670a16   2 months ago   15.2MB
qmcgaw/dns   latest        d2248e987c3d   3 years ago    20MB
root@gl-mt6000:~#

elduds avatar Jan 29 '25 03:01 elduds

Sorry for the massive delay. This is alarming, although could it be because of another container NOT using this dns container perhaps? Or perhaps leaking out via the Docker network local DNS? 🤔

qdm12 avatar Nov 04 '25 22:11 qdm12

I found why:

├── Local DNS middleware:
|   └── Local resolvers:
|       ├── 8.8.8.8:53
|       ├── 8.8.4.4:53
|       ├── [2001:4860:4860::8888]:53
|       └── [2001:4860:4860::8844]:53

This local dns middleware scans your /etc/resolv.conf and extracts nameservers from there to forward local hostnames queries to it, instead of the usual upstream public DNS server. This is so that for example your container names can be resolved in your Docker network.

Now 2 things are wrong:

  1. It should only take DNS servers that are private IP addresses. Fixed in 9ea11dc276ebd96047699b802ca7afae79a01d64
  2. The name tRaCKEr.ToRREnt.Eu.oRG. triggered a code path and the local DNS middleware thought it was a local name. Why? The oldest trick in the book, because of the uppercase 😄 Fixed in c0795f405cc753b1dbdf01f60c0cc09ef8bb9eb6 - that also prompted me to find all usages of the query name and lowercase it. I expected the low level dns library to do it, but it seems it doesn't do it!

qdm12 avatar Nov 05 '25 02:11 qdm12

1c1787b8d02c84073af32a6582303ca6c526397e now converts all query names to lowercase, just in case ™️ If there is ever a usecase for uppercase/mixed-case then we'll adjust!

qdm12 avatar Nov 05 '25 02:11 qdm12