HTTP(S) connections (without DNS) to certain hosts are blocked (by ISP?)

[yuukiAme]

Creating a bug report/issue

• [x] I have searched the existing open and closed issues

Required Information

• DietPi version | cat /boot/dietpi/.version

root@DietPi:~# cat /boot/dietpi/.version
cat /boot/dietpi/.version
G_DIETPI_VERSION_CORE=9
G_DIETPI_VERSION_SUB=3
G_DIETPI_VERSION_RC=0
G_GITBRANCH='master'
G_GITOWNER='MichaIng'

• Distro version | echo $G_DISTRO_NAME $G_RASPBIAN

root@DietPi:~# echo $G_DISTRO_NAME $G_RASPBIAN
bookworm 0

• Kernel version | uname -a

root@DietPi:~# uname -a
Linux DietPi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux

• SBC model | echo $G_HW_MODEL_NAME or (EG: RPi3)

root@DietPi:~# echo $G_HW_MODEL_NAME
RPi 4 Model B (aarch64)

• Power supply used | (EG: 5V 1A RAVpower)

Raspberry Pi Power supply 5V 3A

• SD card used | (EG: SanDisk ultra)

SanDisk Ultra 64G

Additional Information (if applicable)

• Software title

PiVPN

• Was the software title installed freshly or updated/migrated?

Freshly installed

• Can this issue be replicated on a fresh installation of DietPi?

Not sure.

• Bug report ID | echo $G_HW_UUID

root@DietPi:~# echo $G_HW_UUID
c16aa61b-b23a-4d7c-adcd-91591412a10b

Steps to reproduce

1. install Dietpi image on RPi

2. apt update && apt upgrade -y

3. dietpi-launcher

4. install Pi-hole, Unbound, activate DoT with the conf from Dietpi Docs

5. install pivpn, wireguard, setup DNS support with Pi-Hole ,add user, scan QR code

6. add conf file to a device (e.g: an Android phone) to connect to pivpn OUTSIDE of the network

7. check for the websites that is KNOWN to be blocked like 18+ sites, see if it returns query on unbound, load the content on the browser within my Local Network, meaning the DoT is working because my DNS query is encrypted and returned to Unbound with no issue.

8. Turn on Wireguard VPN on the device in Step 6

9. check if the known blocked website works and contents are loaded in the browser.

Expected behaviour

It should load all the contents of the website in the browser.

Actual behaviour

The browser will not load any content and return an error like "Unable to connect". Like an unencrypt query was rejected by the ISP DNS.

Extra details

I can use Pi-Hole with Unbound (activated DoT (DNS over TLS)). I can browse normally to any websites I want other than the block list inside Pi-Hole. My country required ISP to blocks certain websites from their own DNS. So ISP will allow your network to query the website, but it will reject the content of the website.

Help me to troubleshoot if PiVPN is leaking the query because when I access my network over PiVPN, it behaves exactly like when I access the blocked websites WITHOUT Pi-Hole and Unbound DoT which mean my query to unencrypted and intercept by my ISP.

My question is:

How do I check/troubleshoot PiVPN whether it is encrypting my query to Pi-Hole and Unbound on port 5335 or leaving the query unencrypted like port 53?

[Joulinar]

We should clarify a few basic things at the beginning.

When using a VPN such as Wireguard (PiVPN is only an interface on the server for managing clients), all traffic between the VPN client and VPN server is always encrypted. It doesn't matter what it is. Therefore, all requests, including DNS, between your mobile phone and your server are encrypted.

Communication between PiHole and Unbound is always unencrypted. However, this does not matter, as this only takes place on the server.

DNS requests between Unbound and the upstream DoT server, on the other hand, are encrypted again via TLS protocol.

Normally there should be no difference for DNS requests between a mobile phone via VPN or a client in the local network, as all requests first go to the PiHole and then to Unbound.

The chain actually looks like this: Client > (VPN) > PiHole > Unbound > Upstream DNS (via DoT)

Translated with DeepL.com (free version)

[yuukiAme]

We should clarify a few basic things at the beginning.

When using a VPN such as Wireguard (PiVPN is only an interface on the server for managing clients), all traffic between the VPN client and VPN server is always encrypted. It doesn't matter what it is. Therefore, all requests, including DNS, between your mobile phone and your server are encrypted.

Communication between PiHole and Unbound is always unencrypted. However, this does not matter, as this only takes place on the server.

DNS requests between Unbound and the upstream DoT server, on the other hand, are encrypted again via TLS protocol.

Normally there should be no difference for DNS requests between a mobile phone via VPN or a client in the local network, as all requests first go to the PiHole and then to Unbound.

The chain actually looks like this: Client > (VPN) > PiHole > Unbound > Upstream DNS (via DoT)

Translated with DeepL.com (free version)

I understand all that. Thanks for the clarification. So my current VPN traffic should be encrypted. Only the traffics from Pi-hole and Unbound is unencrypted.

Then what exactly can I do to test and find out why my query fails to certain websites in my VPN? Yet my traffic in local network works as intended.

[Joulinar]

Check whether all traffic from the VPN client to the VPN server goes through the tunnel or whether only a split tunnel is active. AllowedIPs = 0.0.0.0/0 should be set in the Wireguard client configuration.

[MichaIng]

WireGuard (clients) have a setting DNS to enforce a different DNS server to query, which makes sense since a local one might not be reachable when traffic is forced through the tunnel. If your PiVPN is running on the same server which hosts Pi-hole, use the WireGuard-internal VPN server IP with this setting.

Note that this is only a client-side setting, same as whether all traffic is forced through the VPN, requests to the server-side LAN or the server itself only. So it is all about how the clients are configured.

Ah, and don't forget to allow "All origins" in Pi-hole for DNS queries. By default it allows queries only from the LAN/main interface of the server, hence blocks queries form any VPN interface.

EDIT: Use AllowedIPs = 0.0.0.0/0, ::/0 at best (if you really want all traffic to be passed through the VPN) to assure IPv6 is as well forced through the tunnel. Else, if IPv6 is enabled on the client, all traffic to IPv6-enabled hosts would bypass the VPN.

[yuukiAme]

Check whether all traffic from the VPN client to the VPN server goes through the tunnel or whether only a split tunnel is active. AllowedIPs = 0.0.0.0/0 should be set in the Wireguard client configuration.

From my Wireguard client conf

[Interface]
PrivateKey = iCHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHcHE=
Address = 10.9.82.2/24
DNS = 10.9.82.1

[Peer]
PublicKey = FkPLcT0Nyt3JJJJJJJJJ+TMJAPwxEaxYOrcNTs=
PresharedKey = P292QilxCCCCCCCCCCCCCCCCCru4pkHzzPqCpK4=
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = example.domain.com:51820

Ignore the PrivateKey , PublicKey, and PresharedKey, I changed to hide the actual value. So I definitely have AllowedIPs = 0.0.0.0/0, ::/0

[yuukiAme]

WireGuard (clients) have a setting DNS to enforce a different DNS server to query, which makes sense since a local one might not be reachable when traffic is forced through the tunnel. If your PiVPN is running on the same server which hosts Pi-hole, use the WireGuard-internal VPN server IP with this setting.

Note that this is only a client-side setting, same as whether all traffic is forced through the VPN, requests to the server-side LAN or the server itself only. So it is all about how the clients are configured.

Ah, and don't forget to allow "All origins" in Pi-hole for DNS queries. By default it allows queries only from the LAN/main interface of the server, hence blocks queries form any VPN interface.

EDIT: Use AllowedIPs = 0.0.0.0/0, ::/0 at best (if you really want all traffic to be passed through the VPN) to assure IPv6 is as well forced through the tunnel. Else, if IPv6 is enabled on the client, all traffic to IPv6-enabled hosts would bypass the VPN.

It's a full tunnel, which allows access to everything within the network. I can use http://pi.hole/admin, even SSH into any VM or server I have in my local network, e.g. Nginx Proxy and Portainer, etc...

I checked the public IP address and it's the same as the one from the Local Network, not the coffee shop public IP address or LTE Public IP. I also checked the dnsleaktest and they all reported that it is using Cloudflare DNS which is my upstream DNS inside of Unbound.

[MichaIng]

I also checked the dnsleaktest and they all reported that it is using Cloudflare DNS which is my upstream DNS inside of Unbound.

Hmm, then I do not understand what the issue is, when clients do in fact use that Pi-hole instance as DNS server through the VPN. If those are Linux/UNIX clients, can you check /etc/resolv.conf, whether 10.9.82.1 is really the only namesever entry?

[yuukiAme]

can you check /etc/resolv.conf

root@DietPi:~# cat /etc/resolv.conf
nameserver 1.1.1.1
nameserver 1.0.0.1

[MichaIng]

I mean on the client(s).

[yuukiAme]

I mean on the client(s).

RIP. it's an iPhone. I also have an Android phone with Termux installed. Give me a few minutes to setup another pivpn client on that Android device.

[yuukiAme]

The Termux doesn't work with cat /etc/resolv.conf. Nevermind.

Any other methods?

[Joulinar]

From your mobile phone, you should see the DNS queries inside the PiHole log. Do you? Even for the website in question. If this is the case, DNS resolution is working fine.

[MichaIng]

Yes, especially check whether you see a query log entry for those cases, where a website is blocked, while you expect it to be accessible thanks to the VPN DNS.

Ah, and also check whether the related browser has DoH enabled, which would bypass the system DNS and VPN.

[yuukiAme]

From your mobile phone, you should see the DNS queries inside the PiHole log. Do you? Even for the website in question. If this is the case, DNS resolution is working fine.

@Joulinar Yes, I can access Pi-hole over the VPN, use the http://pi.hole/admin/, and log in the admin panel, see the query log with my client name called wireguard.pivpn. DNS resolution is definitely working. I even check with SSH into the pi (over VPN), use dig to check Unbound issues. I even checked the Pi-hole > Tools > Search Adlists, to make sure none of my blocklists blocked the exact domain I want to test.

@MichaIng For sure, the DoH, I always check that one off within the browser, on Android 13 and later, inside its Settings, it has Private DNS. I made sure to turn that off first, not even Automatic.

[Joulinar]

Question was if you see inside PiHole log the DNS request for the website not working. If this is the case, DNS resolution is not your problem.

[yuukiAme]

Question was if you see inside PiHole log the DNS request for the website not working. If this is the case, DNS resolution is not your problem.

@Joulinar I used another Android device named ai, connected over VPN and test out an 18+ site

Pi-hole logged the query

Screenshot on my Tablet device named ai- ai.pivpn

On my PC, with pornhub.com cache, no VPN

[Joulinar]

yes DNS resolution is working, so it's something else. Like the browser escaping the VPN tunnel

[yuukiAme]

Like the browser escaping the VPN tunnel

And exactly like how my browser behaves when my query was unencrypted (without Unbound DoT) using 1.1.1.1 at port 53 so my ISP DNS rejects the query. So that's why I'm stuck here. Wondering if PiVPN is "leaking" the query. But I know it shouldn't be.

[Joulinar]

It's not a DNS issue as you see the query inside PiHole. As well it is forward to Unbound according your PiHole screen shot. It's not the DNS query escaping but somehow the browser not using to VPN to access the website.

[yuukiAme]

I thought it was a fluke with Firefox. So I installed Vivaldi Browser for Android. I went into Vivaldi Settings > Privacy and security > and disabled use secure DNS : Off

I deliberately clicked on a Vivaldi bookmark added that I know for sure is in the blocklist. It blocked it successfully. But I tried with pornhub.com and the content is never delivered like Firefox.

I think you're right, @Joulinar .

the browser not using to VPN to access the website.

The browser is clearly using the Pi-hole DNS inside the VPN, but the domains blocked by my country is not working inside it.

[MichaIng]

Can you check whether the IP address pornhub.com is resolved to, is the same when accessing from your PC (within your LAN) and when accessing from the VPN client? If both get the same IP address, and all traffic of the client is passed through the VPN, I have no idea how the client could behave any differently. The browser should not be able to bypass the VPN, at least not when using accessing public/routable IP address. Other than DNS, network traffic/routes are enforced at system level.

And you are sure that Unbound is configured to use DoT? Compare with our docs: https://dietpi.com/docs/software/dns_servers/#__tabbed_2_5

Btw, on your PC pornhub.com works since you use DoH there?

[yuukiAme]

Can you check whether the IP address pornhub.com is resolved to, is the same when accessing from your PC (within your LAN) and when accessing from the VPN client?

on PC - LAN Result: Working as intended

on Tablet (Android) - VPN Result : Blocked

It's using the same IP address. The Cache is from the first query from PC.

And you are sure that Unbound is configured to use DoT? Compare with our docs: https://dietpi.com/docs/software/dns_servers/#__tabbed_2_5

Yes. I was troubleshooting another problem with domain end in .site with NLnetLabs/Unbound yesterday on Github. Issue here

Btw, on your PC pornhub.com works since you use DoH there?

Yup. It works. Same with other 18+ sites and news sites that my countries blocked.

[MichaIng]

Interesting, the LAN client seems to send a DoH query to Pi-hole, which is forwarded to Unbound, if I understand the "HTTPS" query type correctly?

However, so the A type query is answered as intended, works for the LAN client, is cached as intended, and hence answered from cache to the VPN client, but does not work there.

Confusing, indeed. The Android tablet is currently also located in your LAN, just using the VPN (unnecessarily, for testing), right? So that really only enabling the VPN makes the difference?

[yuukiAme]

Interesting, the LAN client seems to send a DoH query to Pi-hole, which is forwarded to Unbound, if I understand the "HTTPS" query type correctly?

I don't have DoH enabled for this network at all. Only Pi-Hole with Unbound DoT. I understand the difference between them. Like how Adguard-DNS / Adguard Home using DNS-over-HTTPS with their SSL cert and privatekey in Encryption Settings.

The Android tablet is currently also located in your LAN, just using the VPN (unnecessarily, for testing), right? So that really only enabling the VPN makes the difference?

I have 3 different ways of connecting to the internet.

1. Network number 1 have Pi-Hole with Unbound DoT and PiVPN
2. Network number 2 for my family uses, have no DNS server, or anything. Just ISP modem/router combo. So ISP can see my unencrypted query easily and reject it.
3. 4G/LTE from my phones.

So I use the Tablet connected to Network number 2 and turn on Wireguard app to connect to PiVPN and it connected to Network number 1 and I open pornhub.com in my Firefox browser and it's blocked.

I tried the exact same thing with 4G/LTE network as above. Same result.

Which led me to the Title of the issue "Leak DNS query over VPN connection". I know it isn't true. That's why I'm asking for help to find faults in my understanding with PiVPN and disprove the title.

EDIT: changed # to number because Github pinged other issue numbers.

[MichaIng]

PiVPN means a (regular) WireGuard server in your case. Might be important when you check for answers elsewhere. PiVPN is only the installer, and the CLI to create client configs, otherwise it has no effect on the VPN.

With AllowedIPs = 0.0.0.0/0, ::/0, leaking the DNS query or anything else should not be possible. This creates a route which forces every routable request through the VPN. Only requests to local/LAN range IP addresses can then bypass the VPN. A killswitch can be used to prevent that as well. So I think, whatever happens, it happens at the VPN server/LAN side. We see the DNS query at Pi-hole, so that does not seem to be the issue. Probably traffic from the pornhub.com IP is blocked, not the DNS query, but that should apply for any other request from LAN the same way.

Maybe best to use tcpdump at the server, so see which requests are coming and going from the VPN client. Also, when you temporarily switch to using Pi-hole as DNS resolver for the VPN/Pi-hole server itself (/etc/resolv.conf), does accessing pornhub.com via curl work?

[yuukiAme]

when you temporarily switch to using Pi-hole as DNS resolver for the VPN/Pi-hole server itself (/etc/resolv.conf), does accessing pornhub.com via curl work?

output

root@DietPi:~# nslookup pornhub.com
Server:         1.1.1.1
Address:        1.1.1.1#53

Non-authoritative answer:
Name:   pornhub.com
Address: 66.254.114.41

follow up output with curl

root@DietPi:~# curl pornhub.com
<!DOCTYPE html>
<html>
<header><title>404 Page</title></header>
<body>
<font size="6"><b>We're sorry but the page you're looking for could not be found!!!!!</b></font>
</body>
</html>

which is expected because of the transparent DNS query.

I will do the tcpdump later because this will take a while to check.

[yuukiAme]

Wait. Should I change the Static DNS in Dietpi-config from Cloudflare DNS to to Pi-Hole with unbound DoT itselfinstead?

[Joulinar]

usually, DNS setting in DietPi itself has no relation to PiHole. And best practice is not to change to a local DNS server like PiHole or AGH.

[yuukiAme]

Background: I brought my Macbook Air to the coffee shop. It took a while to setup the Wireguard because of the newer Wireguard cannot be installed from App Store. I had to deal with brew, Wireguard-tools and, wireguard-go

Macbook output /etc/resolv.conf

mac@192 ~ % cat /etc/resolv.conf 
#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
#   scutil --dns
#
# SEE ALSO
#   dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
nameserver 10.9.82.1

Macbook output on curl pornhub.com

mac@192 ~ % curl pornhub.com
<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>

Macbook requests after using curl pornhub.com being caputure with tcpdump on the Server (Pi-hole)

REMOVED

EDIT: I will remove the tcpdump.txt in a few hours. Because it contains a lot of public IP address that could reveal my location.

[MichaIng]

which is expected because of the transparent DNS query.

I am not sure about that: The DNS query and the actual connection to the resolved host are two different things. Usually, sites are blocked by ISP by spoofing the DNS response with none/invalid. The IP you get from the transparent DNS response however is correct. It seems that the response from the pornhub host itself is intercepted. curl pornhub.com should return a 301 to HTTPS. Can you test this:

curl -I https://www.pornhub.com/

Your tcpdump also shows the response from reflectededge.reflected.net.http, which is indeed the CDN pornhub uses, i.e. the same happens when I do a successful request here.

HTTPS responses cannot be spoofed without the client at least warning about a wrong certificate. So if above HTTPS call fails the same way for you, it is pornhub themselves, or their CDN, who deny to send you data, not the ISP.

The other 17.* IP which appears is from Apple, so I guess this is unrelated?