AdGuardDNS
AdGuardDNS copied to clipboard
AdguardTeam
Akamai and EDNS Client Subnet issues
Issue Details
$ CLASS=CH RRTYPE="TXT" dnslookup id.server tls://dns.adguard.com
dnslookup v1.9.1
dnslookup result (elapsed 276.594529ms):
;; opcode: QUERY, status: NXDOMAIN, id: 60293
;; flags: qr rd ra; QUERY: 1, ANSWER: 0,
AUTHORITY: 1, ADDITIONAL: 6
;; QUESTION SECTION:
;id.server. CH TXT ;;
AUTHORITY SECTION:
. 1237 IN SOA a.root-servers.net.
nstld.verisign-grs.com. 2024012202 1800 900
604800 86400
;; ADDITIONAL SECTION:
client-ip.adguard-dns.com. 3600 CH TXT "14.226.99.x"
server-ip.adguard-dns.com. 3600 CH TXT "94.140.14.14"
country.adguard-dns.com. 3600 CH TXT "VN"
asn.adguard-dns.com. 3600 CH TXT "45899"
subdivision.adguard-dns.com. 3600 CH TXT "HP" resp.res-
type.adguard-dns.com. 3600 CH TXT "normal"
Expected Behavior
When you are in the North of Vietnam, your ECS should be set to ones in the North, because the client should connect to CDNs in Hanoi/HCM/Hongkong for shorter distances and latency, instead of Singapore.
Actual Behavior
$ RRTYPE=TXT dnslookup o-o.myaddr.l.google.com tls://dns.adguard.com
dnslookup v1.9.1
dnslookup result (elapsed 285.909138ms):
;; opcode: QUERY, status: NOERROR, id: 18760
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;o-
o.myaddr.l.google.com. IN TXT
;; ANSWER SECTION:
o-o.myaddr.l.google.com. 60 IN TXT "edns0-client-subnet 222.255.238.0/24"
o-o.myaddr.l.google.com. 60 IN TXT "138.199.46.250"
$ ipinfo.sh 222.255.238.0 [11:14:28]
{
"ip": "222.255.238.0",
"hostname": "static.vnpt.vn",
"city": "Ho Chi Minh City",
"region": "Ho Chi Minh",
"country": "VN",
"loc": "10.8230,106.6296",
"org": "AS45899 VNPT Corp",
"postal": "71606",
"timezone": "Asia/Ho_Chi_Minh"
}
From a screenshot in the video below, I see subdivision is applied to CN, IN... but not to VN. Is it right?
https://github.com/AdguardTeam/AdGuardDNS/issues/659#issuecomment-1904324548
Adguard DNS:
$ dig starling-oversea.byteoversea.com @dns.adguard.com +short
starling-oversea.byteoversea.com.ttdns2.com.
starling-oversea.byteoversea.com.edgekey.net.
e11942.a.akamaiedge.net.
23.215.7.32 23.215.7.28
23.215.7.19 23.215.7.16
23.215.7.22 23.215.7.23
23.215.7.20 23.215.7.25
23.215.7.18
$ ping -c3 23.215.7.15 [11:49:13]
PING 23.215.7.15 (23.215.7.15) 56(84) bytes of data.
64 bytes from 23.215.7.15: icmp_seq=1 ttl=57 time=47.5 ms
64 bytes from 23.215.7.15: icmp_seq=2 ttl=57 time=46.1 ms
64 bytes from 23.215.7.15: icmp_seq=3 ttl=57 time=48.9 ms
--- 23.215.7.15 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 46.087/47.489/48.892/1.145 ms
Google DNS:
$ dig starling-oversea.byteoversea.com @dns.google +short
starling-oversea.byteoversea.com.ttdns2.com.
starling-oversea.byteoversea.com.edgekey.net.
e11942.a.akamaiedge.net.
113.171.10.152 113.171.10.144
113.171.10.162 113.171.10.160
113.171.10.155 113.171.10.154
113.171.10.146 113.171.10.139
113.171.10.145
$ ping -c3 113.171.10.155 [11:49:33]
PING 113.171.10.155 (113.171.10.155) 56(84) bytes of data.
64 bytes from 113.171.10.155: icmp_seq=1 ttl=58 time=6.05 ms
64 bytes from 113.171.10.155: icmp_seq=2 ttl=58 time=4.58 ms
64 bytes from 113.171.10.155: icmp_seq=3 ttl=58 time=3.22 ms
--- 113.171.10.155 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.220/4.618/6.052/1.156 ms
Right, the subdivision logic at this moment is enabled for 4 countries only: US, Russia, China and India.
Interesting data, thank you! We'll experiment with it a bit, I am not yet sure that it should be solved. We should first figure out why Akamai returns Singapore addresses for this subnet, it's unclear from the BGP information what's so different with this prefix compared to other prefixes announced by AS45899.
We should first figure out why Akamai returns Singapore addresses for this subnet
Because that ECS belongs to HCM which is near Singapore? I guess a ECS in HN should be better. Since my subnet is in Hanoi (the north of Vietnam)
@trmdi HCM?
As I see both subnets (yours and the one that AG DNS uses for ECS) belongs to the same AS45899.
- ECS -
222.255.238.0/24 - Your subnet -
14.226.99.0/24
Maybe you're right, we should check how Akamai nameserver responses change depending on what location the subnet belongs to.
Internal task ID: AGDNS-1881
Just got one more idea. Vietnam has quite a lot of different subdivisions in MaxMind (it uses ISO 3166-2). According to the ISO 3166-2 standard, Vietnam is divided into 58 provinces and 5 municipalities, but for network routing it is probably not that important and it'd be enough to do a "north-south" split.
@trmdi does that sound reasonable to you? If so, how would you split the subdivisions between north/south?
I think we could probably do the same for large countries as well, split subdivisions between west/east/north/south/central in order to improve cache efficiency.
Yes, I think splitting between west/east/north/south/central is fine. Specifically, Vietnam area is S-shaped, narrow and stretched from the north to the south, so I think splitting to north/south (the upper 50% area is the north) is enough for network routing.
Okay it's not that easy.
Something's strange with Akamai. They indicate in responses that they don't rely on ECS:
dig -t a starling-oversea.byteoversea.com. @8.8.8.8 +subnet=14.226.99.0/24
...
CLIENT-SUBNET: 14.226.99.0/24/0
...
Now one more strange thing.
There's another domain that I tested with: store.akamai.steamstatic.com.
For some reason via Google DNS Akamai responds very accurately with Cyprus IP addresses.
dig -t a store.akamai.steamstatic.com. @8.8.8.8
...
;; ANSWER SECTION:
store.akamai.steamstatic.com. 20 IN A 195.14.151.138
store.akamai.steamstatic.com. 20 IN A 195.14.151.155
...
What's strange is that when I pass a custom subnet via dig to Google DNS, it continues to respond with accurate Cyprus IP addresses.
dig -t a store.akamai.steamstatic.com. @8.8.8.8 +subnet=14.226.99.0/24
...
;; ANSWER SECTION:
store.akamai.steamstatic.com. 20 IN A 195.14.151.155
store.akamai.steamstatic.com. 20 IN A 195.14.151.138
...
I honestly do not understand what's going on there and how on earth could Akamai learn about my location considering that I am being routed to Google DNS servers located in Switzerland (and Google DNS does not have any servers in Cyprus anyways).
Checking what Google DNS server is being used and what ECS it sends to authoritative nameservers:
dig -t txt o-o.myaddr.l.google.com. @8.8.8.8 +subnet=14.226.99.0/24
...
;; ANSWER SECTION:
o-o.myaddr.l.google.com. 60 IN TXT "172.253.237.133"
o-o.myaddr.l.google.com. 60 IN TXT "edns0-client-subnet 14.226.99.0/24"
...
Here's what I suspect:
- In the case of Akamai nameservers, Google DNS sends my real subnet in ECS even if I specify the subnet in the query.
- Akamai only accepts ECS from a select list of DNS servers. They accept it from Google DNS and OpenDNS, but not from us or Quad9.
Your suspicious are both correct. They will only respond to ECS from known resolvers with whom they have a contractual relationship, and relationship they have with Akamai does not permit the resolvers to forward ECS from clients.
@vttale thank you for confirming my suspicion!
On a side note, this looks like a weird business decision for Akamai. They should've been interested in speeding up connections for all Internet users, not for a part of them.
Found it: https://community.akamai.com/customers/s/article/What-is-the-ECS-onboarding-agreement-for-No-Client-Side-ECS
Well, our ECS implementation cannot fulfil what they're requiring. I'll try to contact them and maybe find some middle ground.
On a side note, this looks like a weird business decision for Akamai. They should've been interested in speeding up connections for all Internet users, not for a part of them.
From their point of view, their mapping is proprietary information based on extensive measurement of the Internet. They don't want that trove of data to be easily walked through iterative ECS queries.