Incompatible with many DoH providers

[BlackthornYugen]

I'm not sure if I'm doing something wrong... but it seems like most of the servers I try fail to parse in some way or another.

for DOH_TARGET in \
  private.canadianshield.cira.ca/dns-query \
  dns.adguard.com/dns-query \
  doh.libredns.gr/dns-query \
  doh.libredns.gr/ads \
  dns.quad9.net/dns-query \
  doh.opendns.com/dns-query \
  doh.xfinity.com/dns-query \
  doh.powerdns.org \
  doh.ffmuc.net/dns-query
do
  printf "%-40s - " $DOH_TARGET
  doh query analytics.google.com \
    --custom-only \
    --custom-source-url="https://${DOH_TARGET}" \
    --labels \
    --verbose
done

private.canadianshield.cira.ca/dns-query - error: invalid character 'T' looking for beginning of value
dns.adguard.com/dns-query                - error: invalid character 'o' in literal null (expecting 'u')
doh.libredns.gr/dns-query                - error: invalid character 'U' looking for beginning of value
doh.libredns.gr/ads                      - error: invalid character 'U' looking for beginning of value
dns.quad9.net/dns-query                  - error: invalid character 'U' looking for beginning of value
doh.opendns.com/dns-query                - error: invalid character 'M' looking for beginning of value
doh.xfinity.com/dns-query                - error: EOF
doh.powerdns.org                         - error: invalid character 'U' looking for beginning of value
doh.ffmuc.net/dns-query                  - error: invalid character 'U' looking for beginning of value

[jpgpi250]

I have the same problem. I first tried a php client, that was a disaster, errors all over the place... Now I'm trying this client, I'm using servers and urls from this page.

I would expect any URL can at least be used to return an IP address for the server it is hosted on...

going over the servers in the list (I tested a lot of them but NOT all) I found two URLs that produce results:

• https://cloudflare-dns.com/dns-query
• https://doh-de.blahdns.com/dns-query

A lot of the other servers simply return an empty result.

here is my partial test code `

custom="https://cloudflare-dns.com/dns-query"

retryflag=true
while :; do
	RESULT=($(/home/pi/go/bin/doh query ${server} --type A --custom-only --custom-source-url="${custom}" --no-limit --joined))
	if ! grep -q '"Answer":null' <<< "$RESULT" && [ ! -z ${RESULT} ]; then
		IPv4=($(echo $RESULT | jq 'map(.Answer | map(.data)) | flatten | .[]' --raw-output))
		len=${#IPv4[@]}
		echo "$server" | sudo tee -a ${workdir}/${hostfile}.tmp >/dev/null
		for (( i=0; i<$len; i++ )); do
			if ! ipcalc -nb ${IPv4[$i]} | grep -q "INVALID ADDRESS"; then
				echo "${server} has IPv4 ${IPv4[$i]} (from ${custom})"
				echo "${IPv4[$i]}" | sudo tee -a ${workdir}/${IPv4file}.tmp >/dev/null
			fi
		done
		break
	elif [ "${retryflag}" = true ]; then
		retryflag=false
		custom='https://doh-de.blahdns.com/dns-query'
		echo "retry"
	else
		echo "failed"
		break
	fi
done

` looking for IPv6 (--type AAAA) is also a problem (no or empty result on several URL's)

I would like to use a specific (or random) URL to resolve the server's IP address, for example (look for a partial match in the URLlist, but that now appears to be impossible, since most of the URLS return no, or an empty result): `

dohserver=($(grep "${server}" /home/pi/DOH/dohurls.list))
if [ -z "${dohserver}" ]; then
	dohserver=($(shuf -n 1 /home/pi/DOH/dohurls.list))
fi
	
custom=${dohserver}

`

In conclusion, I'm also wondering what I'm doing wrong, since the code appears to be working with two resolvers (see above).

[jpgpi250]

ran another test. apparently, the number of URL's that can resolve, changes if the requested domain varies.

• www.google.com: 22 servers can resolve (list has 60 entries)
• www.raspberrypi.org: 1 server can resolve
• tweakers.net: 20 servers can resolve

#!/bin/bash

j=0
while read list; do
	custom=${list}
	#echo "${custom}"
	RESULT=($(/home/pi/go/bin/doh query raspberrypi.org --type A --custom-only --custom-source-url="${custom}" --no-limit --joined))
	if ! grep -q '"Answer":null' <<< "$RESULT" && [ ! -z ${RESULT} ]; then
		IPv4=($(echo $RESULT | jq 'map(.Answer | map(.data)) | flatten | .[]' --raw-output))
		len=${#IPv4[@]}
		for (( i=0; i<$len; i++ )); do
			if ! ipcalc -nb ${IPv4[$i]} | grep -q "INVALID ADDRESS"; then
				((j++))
				echo "${j}: raspberrypi.org has IPv4 ${IPv4[$i]} (from ${custom})"
			fi
		done
	fi
done < /home/pi/DOH/dohurls.list

[picatz]

👋 Hello @BlackthornYugen, @jpgpi250! Thank you for bringing* this up.

I don't think either of you are doing anything wrong, it's just confusing.

DoH information can be accessed in two ways on some providers:

1. https://dns.google/dns-query – RFC 8484 (GET and POST)
2. https://dns.google/resolve – JSON API (GET)

Some providers are fancy and support both on the same endpoint. Currently doh only supports providers that support the JSON API.

I think in the near-ish future I'm going to use RFC 8484 as the first option, and make the JSON API option available as a command-line flag.

[picatz]

👋 Hello @BlackthornYugen and @jpgpi250, apologies for the long delay in getting this issue resolved.

🎉 https://github.com/picatz/doh/pull/26 should fix this issue, which landed an RFC 8484 based client implementation.

[!IMPORTANT] However, importantly, some of the CLI flags have been removed (--custom-*, --labels) or slightly changed in favor of simpler usage.

The following should be possible:

for DOH_SERVER in \
  private.canadianshield.cira.ca/dns-query \
  dns.adguard.com/dns-query \
  doh.libredns.gr/dns-query \
  doh.libredns.gr/ads \
  dns.quad9.net/dns-query \
  doh.opendns.com/dns-query \
  doh.xfinity.com/dns-query \
  doh.powerdns.org \
  doh.ffmuc.net/dns-query
do
  doh query analytics.google.com \
    --servers="https://${DOH_SERVER}"
done

Please let me know if you run into any issues, thank you!