DoT issue with OpenWRT/Unbound and paid Adguard DNS

[derKief]

Platform

Router

Protocol

DNS-over-TLS

Do you use AdGuard app?

No I don't

Your configuration

OpenWrt 23.05.3 Unbound 1.20.0-1

Traceroute to AdGuard DNS

traceroute to 94.140.14.14 (94.140.14.14), 20 hops max, 46 byte packets 1 192.168.222.1 0.539 ms 2 * 3 81.210.148.64 10.311 ms 4 84.116.197.129 21.541 ms 5 84.116.190.134 16.339 ms 6 80.255.9.77 15.457 ms 7 169.150.195.68 15.526 ms 8 169.150.194.53 15.466 ms 9 94.140.14.14 17.482 ms

Issue Details

activate Free Adguard DNS works activate Paid Adguard DNS doesnt work. deactivate DNSSEC in Unbound works.

so the issue must have to do something with DNSSEC

Expected Behavior

a working configuration of DoT in conjunction with Openwrt/Unbound and the paid Adguard DNS

Actual Behavior

a non working configuration of DoT in conjunction with Openwrt/Unbound and the paid Adguard DNS

Screenshots

Additional Information

Hi, since the update to the most recent version of OpenWRT v23.05.3 (so the assumption) my customized configuration with paid Adguard DNS no longer works. But this exact configuration had worked for months before without any problems. The whole thing must have something to do with DNSSEC. As soon as I deactivate this in Unbound, everything works. I am currently using the free version of Adguard DNS which works without any problems. Also i posted this over at OpenWRT forum but unfortunately no response until now.

Do you have an idea what could be causing this and maybe even a solution?

Thanks in advance and greetings.

[derKief]

updating to OpenWrt 23.05.4 - Service Release doesn't help. At the moment the only option is to either disable DNSSEC or use Adguard free DNS

[derKief]

I found this in Unbound changelog https://github.com/NLnetLabs/unbound/blob/master/doc/Changelog

25 July 2024: Wouter - Add root key 38696 from 2024 for DNSSEC validation. It is added to the default root keys in unbound-anchor. The content can be inspected with unbound-anchor -l.

This also fits roughly with the beginning of the problems. Now I wonder who has to do what here ?

[Chinaski1]

Hello there!

We will check soon if there are any DNSSEC problems on our side.

[derKief]

@Chinaski1 any news on this?

[Chinaski1]

1. I checked, and we haven't encountered any DNSSEC issues on our end
2. First you might want to try configuring DoH (If supported by your router)
3. In case of problems, you should pay attention to unbuond proxy logs. Maybe they can shed some light on what's going on.

[derKief]

meanwhile i tried DoT with Stubby+DNSmasq on OpenWRT. Same issue. As soon as i activate DNSSEC i get this issue. Unfortunately the logs doesn't really help. I running out of ideas.

[derKief]

@Chinaski1 over at Openwrt forum they mentioned that the problem may be related to alternative dnssec root key you are implementing. The problem is annoying now because nobody can or wants to help me. It's the typical ping pong game again

[Chinaski1]

We haven't made any changes with alternative dnssec root key you are implementing. And what will it show when you follow this link, what server are you connected to?

[derKief]

I'm following the instructions on my Dashboard at https://adguard-dns.io/ I cannot test your link because DoT isnt working with private DNS settings Only free public DNS are working.

*Configure DNS-over-TLS Command-line instructions Disable Dnsmasq DNS role or remove it completely optionally replacing its DHCP role with odhcpd.

Install packages

1. opkg update
2. opkg install unbound-daemon ca-certificates LAN clients and the local system should use Unbound as a primary resolver assuming that Dnsmasq is disabled. Web interface If you want to manage the settings using web interface, install the necessary packages.

Install packages

1. opkg update
2. opkg install luci-app-unbound ca-certificates
3. /etc/init.d/rpcd restart Navigate to LuCI → Services → Recursive DNS to configure Unbound. Configure AdGuard DNS-over-TLS
4. uci add unbound zone
5. uci set unbound.@zone[-1].enabled="1"
6. uci set unbound.@zone[-1].fallback="0"
7. uci set unbound.@zone[-1].zone_type="forward_zone"
8. uci add_list unbound.@zone[-1].zone_name="."
9. uci set unbound.@zone[-1].tls_upstream="1"
10. uci set unbound.@zone[-1].tls_index="bebd2xxx.d.adguard-dns.com"
11. uci add_list unbound.@zone[-1].server="94.140.14.xx"
12. uci add_list unbound.@zone[-1].server="94.140.14.xx"
13. uci add_list unbound.@zone[-1].server="2xx0:50c0::ded:ff"
14. uci add_list unbound.@zone[-1].server="2xx0:50c0::dad:ff"
15. uci commit unbound
16. /etc/init.d/unbound restart

[derKief]

still not working. Every party says its not our fault ... we didnt change anything. I also have issues with Adgaurd on my mobile and Adguard support is also not helping there.

I'm currently very unhappy with Adguard and will probably not renew my subscription. Cashing in but no support... that's not acceptable.

[Chinaski1]

Please specify the support request number.

[derKief]

Since no one has been able to help me so far, I have switched to dnsproxy.

[derKief]

As dnsproxy is not really reliable on OpenWRT for me i switched back to old confugartion. This time i tried stubby but still same issue. Followed the guide here ->https://openwrt.org/docs/guide-user/services/dns/dot_dnsmasq_stubby So still the same problem: DNSSEC with Adguard public DNS servers works without issue DNSSEC with Adguard private DNS servers doesnt work.

Need help with this.

Maybe @ameshkov is able to help me with either ...

[derKief]

@Chinaski1 @ameshkov There seem to be parallels to the following ticket. It certainly reads like that.

DNSSEC fails to validate when using my profile

[derKief]

I switched again to dnsproxy+dnsmasq on Openwrt -> https://openwrt.org/docs/guide-user/services/dns/dot_dnsmasq_dnsproxy But I installed dnsmasq full package and then activated dnssec there and it now works at least partially. It is not really usable like this because there are always hangs and long loading/response times

Maybe the following log entries are helpful (I have obscured my config)

Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] Starting dnsproxy v0.67.0 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: cache: disabled Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: starting dns proxy server Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: creating udp server socket 127.0.0.1:5354 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: listening to udp://127.0.0.1:5354 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: creating udp server socket [::1]:5354 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: listening to udp://[::1]:5354 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: creating tcp server socket 127.0.0.1:5354 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: listening to tcp://127.0.0.1:5354 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: creating tcp server socket [::1]:5354 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: listening to tcp://[::1]:5354 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: entering udp listener loop on 127.0.0.1:5354 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: entering udp listener loop on [::1]:5354 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: entering tcp listener loop on 127.0.0.1:5354 Thu Dec 5 12:55:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:04 [info] dnsproxy: entering tcp listener loop on [::1]:5354 Thu Dec 5 12:55:11 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:11 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;com. IN DNSKEY in 1.786474725s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:55:14 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:14 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;box. IN DNSKEY in 4.563327226s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:55:16 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:16 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;com. IN DNSKEY in 2.638639957s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:55:19 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:19 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;com. IN DNSKEY in 5.627714318s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:55:23 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:23 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;net. IN DNSKEY in 1.868534878s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:55:28 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:55:28 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;nl. IN DNSKEY in 1.710602149s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:57:04 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:57:04 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;io. IN DNSKEY in 1.522813828s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:57:16 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:57:16 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;de. IN DNSKEY in 2.332351008s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:57:19 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:57:19 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;de. IN DNSKEY in 1.488108367s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:57:21 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:57:21 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;de. IN DNSKEY in 3.774685403s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:57:23 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:57:23 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;de. IN DNSKEY in 5.500002302s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:57:25 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:57:25 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;de. IN DNSKEY in 7.300611008s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:57:26 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:57:26 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;de. IN DNSKEY in 9.063911801s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:57:28 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:57:28 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;de. IN DNSKEY in 10.804622197s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:57:30 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:57:30 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;de. IN DNSKEY in 12.60295137s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:57:32 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:57:32 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;de. IN DNSKEY in 14.703865934s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx Thu Dec 5 12:58:16 2024 daemon.err dnsproxy[1945]: 2024/12/05 11:58:16 [error] dnsproxy: upstream https://d.adguard-dns.com:443/dns-query/xxxxxxxx failed to exchange ;org. IN DNSKEY in 1.777064977s: expected status 200, got 500 from https://d.adguard-dns.com:443/dns-query/xxxxxxxx

[derKief]

And now I've tested the whole thing again with Unbound. The following log entries might be helpful:

Thu Dec 5 14:47:53 2024 daemon.info unbound: [5981:0] info: resolving org. DNSKEY IN Thu Dec 5 14:47:55 2024 daemon.info unbound: [5981:0] info: Missing DNSKEY RRset in response to DNSKEY query. Thu Dec 5 14:47:56 2024 daemon.info unbound: [5981:0] info: Could not establish a chain of trust to keys for org. DNSKEY IN Thu Dec 5 14:47:59 2024 daemon.info unbound: [5981:0] info: resolving de. DNSKEY IN Thu Dec 5 14:48:01 2024 daemon.info unbound: [5981:0] info: Missing DNSKEY RRset in response to DNSKEY query. Thu Dec 5 14:48:01 2024 daemon.info unbound: [5981:0] info: Could not establish a chain of trust to keys for de. DNSKEY IN Thu Dec 5 14:48:05 2024 daemon.info unbound: [5981:0] info: resolving com. DNSKEY IN Thu Dec 5 14:48:05 2024 daemon.info unbound: [5981:0] info: Missing DNSKEY RRset in response to DNSKEY query. Thu Dec 5 14:48:07 2024 daemon.info unbound: [5981:0] info: resolving net. DNSKEY IN Thu Dec 5 14:48:07 2024 daemon.info unbound: [5981:0] info: Missing DNSKEY RRset in response to DNSKEY query. Thu Dec 5 14:48:07 2024 daemon.info unbound: [5981:0] info: Could not establish a chain of trust to keys for net. DNSKEY IN

[ameshkov]

@derKief hi, sorry for the late reply!

This log is interesting, we at least see the domains that it's trying to query.

Could you please install dnslookup and do a couple of queries using your AdGuard DNS personal address?

RRTYPE=DNSKEY dnslookup org https://d.adguard-dns.com/dns-query/xxxxxxxx

RRTYPE=DNSKEY dnslookup net https://d.adguard-dns.com/dns-query/xxxxxxxx

Here's how it looks like for me:

RRTYPE=DNSKEY dnslookup org https://dns.adguard-dns.com/dns-query
dnslookup 1.11.1
Server: https://dns.adguard-dns.com/dns-query

dnslookup result (elapsed 281.476917ms):
;; opcode: QUERY, status: NOERROR, id: 29390
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;org.	IN	 DNSKEY

;; ANSWER SECTION:
org.	1933	IN	DNSKEY	256 3 8 AwEAAbLYnI4VW05DKivdc/FPN+6rhjMUx87qb03sxmoYQiO9SLqWuzplcyMRtW0k8rcEWa+WiIEoAFqv+fARmWJmqc4rWlZPe/YXcEy6tb3LnMu9ohb+R9GiJw9B9AMAU6FEEjhoXu/wPwcLKu0ExQkVLjDEilyKX1NmN+Ek7vBK2G+5
org.	1933	IN	DNSKEY	256 3 8 AwEAAdim1OsPrczOdInedsci34awEy8Y2z3wk/vBneTpBcMqZ/RMblR628sT7w1DmLTqtVGxJZseiWV4ZPopdncraq3psjtMZ09Aacdho5aToAlRGkemALvc41I0fxt1oORr4UxHjx0oYWgpVgFfn8WrolxfJs1HojL4KvHtGbOOLrod
org.	1933	IN	DNSKEY	257 3 8 AwEAAexZJ/1wfyNCxNPrTZizaG7UlibGhP+AyogR6bqjptKweEgE4gD8GxRQJkt+Fn5pCoNqzmm1ZnEoKqvm93uOYtbKkYQDGH+W69J66MSKpgIyS+mT/4iaXn+lpb5o99l/sf7lHMa975O/fqN6aPUll4hUbN2T1LHv6HzQuQCtNRJA8jHGwX5q0NMmh2Z+yaG6B9cISerje9l5L+ID2ydJ6zXquYteoIUvX2xzqnXCdHPSvD+oL6R/weW+tztdFS1hok/1z3tn5NzmcaOLll9nXniCozEpLFEGPswyvtphWgCYhI8bBTqhUsIwfIwLSBQTEg2oCX7sS5CbXg44OqwhIW8=

[derKief]

@ameshkov shall i do that from the router itself or from a device in local network ? But its only possible to do that with DNSSEC disabled in Unbound.

This is on a LAN device with DNSSEC disabled:

[ameshkov]

On Windows you should do it like that:

set RRTYPE=DNSKEY

dnslookup.exe org <youraddress>

Also, could you please tell me what you see when you open https://dns.adguard-dns.com/info.txt ?

[ameshkov]

@asdfjkluiop @derKief I've finally reproduced the issue, should be easier for us to figure it out now.

[ameshkov]

Argh, I figured it out, this is due to "Block known scanners" in the Access settings.

One of the rules there was too wide and it was blocking all DNSKEY queries to top-level-domains. I removed that rule now, the issue should go away in about an hour or two.

[derKief]

thats good news... will report back

[derKief]

@ameshkov its fixed now ... thanks a lot.

[asdfjkluiop]

Hmmmm, still broken for me curiously. Attempting to delv anything against my adguard profile still gives a broken trust chain

[ameshkov]

@asdfjkluiop what website are you trying to check? Or is it happening with any DNSSEC-enabled website?

Could you please run delv with -d 3 so that we could see where is the problem?

And one more question: does it help to disable Access settings for your profile?

[asdfjkluiop]

@ameshkov weird...maybe it just took a really long time to fully apply because everything is just working now. What I was doing was delv aaaa cloudflare.com @myprofile but I just went back to test it right now and it's fine even with access settings enabled.