cloudflared

cloudflared copied to clipboard

I am not sure how to diagnose the issue but hoping someone can help. Not overly sure if it is an issue with Clouflared.

I have successfully setup Cloudflared to act as a DNS server and using it with Pi-Hole. I have manually specified my DNS on a laptop and that works perfectly.

Unfortunately is I change my DNS in the router Cloudflared stops resolving DNS. I have double checked this by connecting using SSH and manually attempting a DNS query and nothing is returned. It stopped working immediately after changing the router to hand out the DNS server.

The steps taken to setup closely follow: https://scotthelme.co.uk/securing-dns-across-all-of-my-devices-with-pihole-dns-over-https-1-1-1-1/

Pull request #24 fixes this and gives more useful errors

I also followed Scott instructions and the version I installed was cloudflared version 2018.4.8 (built 2018-04-26-1817 UTC). I had a similar problem but I can't understand if it's the same: at the beginning it was working fine, then everything got very slow and cloudflared was taking up to 95% of the CPU, slowing down even the PiHole dashboard. I tried a reboot, I tried to manually start it but nothing changed/improved. At the end I decided to deactivate it and switch back to a normal DNS.

Question: do you know how can I get a copy of the cloudflared version 2018.4.5 (built 2018-04-09-2155 UTC) binary, which is the same version used by Scott, just to understand if I'm getting the same problem with that version or if it could be a regression in the version I installed.

Cheers

Just for reference, I started seeing this in the terminal (I cut a few lines to make it shorter):

pi@raspberrypi:~/argo-tunnel $ sudo ./cloudflared proxy-dns --port 54 --upstream https://1.1.1.1/.well-known/dns-query --upstream https://1.0.0.1/.well-known/dns-query
INFO[0000] Adding DNS upstream                           url="https://1.1.1.1/.well-known/dns-query"
INFO[0000] Adding DNS upstream                           url="https://1.0.0.1/.well-known/dns-query"
INFO[0000] Starting DNS over HTTPS proxy server          addr="dns://localhost:54"
INFO[0000] Starting metrics server                       addr="127.0.0.1:35905"
ERRO[1993] failed to connect to an HTTPS backend "https://1.1.1.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.1.1.1/.well-known/dns-query: read tcp 192.168.1.109:58962->1.1.1.1:443: read: connection timed out"
ERRO[1993] failed to connect to an HTTPS backend "https://1.1.1.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.1.1.1/.well-known/dns-query: read tcp 192.168.1.109:58962->1.1.1.1:443: read: connection timed out"
ERRO[1993] failed to connect to an HTTPS backend "https://1.1.1.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.1.1.1/.well-known/dns-query: read tcp 192.168.1.109:58962->1.1.1.1:443: read: connection timed out"
ERRO[1993] failed to connect to an HTTPS backend "https://1.1.1.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.1.1.1/.well-known/dns-query: read tcp 192.168.1.109:58962->1.1.1.1:443: read: connection timed out"
ERRO[2062] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2063] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2064] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2064] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"

I do not use Scott's stuff but @andreagrandi got exactly the same problem. https://github.com/cloudflare/cloudflared/issues/23#issuecomment-387000967 Worked fine until yesterday.

Because there is no possibility to download the previous release, it is difficult to debug out where the problem is.

@andreagrandi If you can find the previous version. Please let me know.

@xetorixik sorry, but I don't understand: are you experiencing my same problem or not? Cheers

@andreagrandi Why do you specify url with .well-known in it? Shouldn't it be https://1.1.1.1/dns-query and https://1.0.0.1/dns-query? I don't see any mention of it here

hmm, good question @mcspr I was curious about the URL when I was following the instructions. Not sure if it would cause the fault but worth fixing.

@mcspr I saw that url here https://scotthelme.co.uk/securing-dns-across-all-of-my-devices-with-pihole-dns-over-https-1-1-1-1/ which one should I use instead? thanks

@andreagrandi Urls that I mentioned, or just do not use --upstream ... because they are builtin as default choices: https://github.com/cloudflare/cloudflared/blob/9135a4837c16a8f1f0dc4073d339db0a03d6579f/cmd/cloudflared/main.go#L264-L269

@AlexaBible You can test that curl 'https://1.1.1.1/.well-known/dns-query?ct=application/dns-json&name=cloudflare-dns.com&?type=A' does not work.

I just removed the --upstream parameters and it seems to be running fine! I will keep an eye on this and let you know if I still get high CPU usage. Thanks for helping!

False alarm :( previously I was misled because I had my VPN on and it bypass the system DNS. Disconnecting the VPN (and using pihole DNS) the cloudflared doesn't work at all for me. I can't see any error in the console but no address is being resolved. I reverted using a fix DNS and deactivated cloudflared.

I'm getting this issue with Cloudflared version 2018-7.2 on a Raspberry Pi 2 Model B. Did anyone resolve this somehow? Experiencing the same random failures + need to reboot to get it working again. (Undesirable if I'm not home and can't reboot the Pi for other users)

Personally, I reinstalled but followed the official guide. I then just set the dns in the GUI. From then on it worked for me.

@AlexaBible thanks for the quick response—can link me to the guide?

@cvocvo I believe he's referring to this page.

I have also encountered the issue, I am yet to attempt Alexa's suggestion but I will post an update here if it works

Also getting this problem sometimes, seems to happen when I reboot my router and the internet connection is not available (running a pihole with cloudflared). Even when the router is running again after the reboot the cloudflared client keeps spitting out errors, a restart of cloudflared seems to fix this... Is there a more elegant way to fix this?

Oct 23 18:38:33 raspberrypi cloudflared[360]: time="2018-10-23T18:38:33+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 23 18:38:33 raspberrypi cloudflared[360]: time="2018-10-23T18:38:33+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

cloudflared --version cloudflared version 2018.10.3 (built 2018-10-10-2045 UTC)

In my case, it stopped when my laptop connected to a VPN network. I must restart it manually. Error messages below:

$ systemctl status cloudflared.service
...
...msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
...msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"

FWIW, this seems specific to cloudflared rather than one of the Go packages. I've been using dnscrypt-proxy (also written in Go) as an alternative and it has been running flawlessly.

I tried dnscrypt-proxy. It was nice for a while when waking from sleep, recovering from screen lock, switching between LAN and VPN. But sometimes it has been same trouble, especially when switching between LAN and VPN for a long interval. I must restart it also. I'm puzzled why it happen.

Getting these errors on my pi too after an internet reboot or drop-out and reconnect,

failed to connect to an HTTPS backend "https://1.1.1.1/dns-query"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Strange thing is, I used this on another RPI also running cloudflared too and it doesnt not have the same issue, it seems to noticed when the internet drops out and the HTTPS connection re-establishes fine. Configuration of the proxy is exactly the same.

After a manual service restart, things are up and running as normal.

@callifo I would be curious to re-try again the whole setup with a more recent version of cloudflared. Last time I tried it was May 2018.

Yeah I'm running 2018.10.0 which I think is still the latest, and it still seems to behave the same way you described. A lot of issues all around the same date May-ish this year but not much since.

Was hoping for a fix but its been going on for a while. Might have to go low brow and use bash to detect when google.com stops resolving and reboot the daemon.

I've compiled version 2018.12.1 for mipsle and I was happy that it was working on my Ubiquiti ERX, but after multiple complaints that DNS resolution stopped working I had to remove cloudflared and just do plain DNS to 1.1.1.1 :(

I get the same "failed to connect to an HTTPS backend" and also netstat reports multiple sockets in CLOSE_WAIT state.

Retested 2018.12.1 and still having the same issue. Fortunately my internet stays up for months at a time so its not too inconvenient.

Its really odd, as I have two RPI's running it in different locations and both are configured the same with cloudflared, and only the local one has the issue. The other can handle internet drops without a problem.

I have also been having this issue with 2018.12.1 when the internet connection drops. I have to restart the cloudflared service for it to start resolving again on my RPi.

cloudflared[2809]: time="2019-01-21T18:55:01Z" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https:

I'm also getting lots of this errors:

ERRO[0614] failed to connect to an HTTPS backend "https://1.1.1.1/dns-query"  error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: read tcp 192.168.86.113:50946->1.1.1.1:443: read: connection reset by peer"

I'm on mac installed via brew running VERSION: 2018.12.1 (built 2018-12-11-2047 UTC)

Having the same issue as everybody else:

pi@pihole:~ $ systemctl status cloudflared
● cloudflared.service - cloudflared DNS over HTTPS proxy
   Loaded: loaded (/lib/systemd/system/cloudflared.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2019-02-09 21:34:04 CET; 1min 4s ago
 Main PID: 1433 (cloudflared)
   CGroup: /system.slice/cloudflared.service
           └─1433 /usr/local/bin/cloudflared proxy-dns --port 5053 --upstream https://1.1.1.1/dns-query --upstream https://1.0.0.1/dns-query


Feb 09 21:31:38 pihole cloudflared[346]: time="2019-02-09T21:31:38+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

~~I am running quite old version as it seems to be the last one compatible with Raspberry 1st gen.~~

pi@ pihole:~ $ cloudflared -v
cloudflared version 2018.7.2 (built 2018-07-13-1701 UTC)

Update: I just built from the latest source to get around the segfault in the official binary, still getting the error.

+1

Feb 13 18:57:13 localhost cloudflared[5807]: time="2019-02-13T18:57:13+11:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"

cloudflared version 2018.12.1 (built 2018-12-11-2030 UTC)

Same thing here, it works but when I reboot my Raspberry (Pi-Hole) I also get the mentioned errors: level=error msg="failed to connect to an HTTPS backend \"[https://1.1.1.1/dns-query\](https://1.1.1.1/dns-query)"" error="failed to perform an HTTPS request etcetcetc

When I restart cloudflared everything is fine again, really annoying (bug?) :'(

I'm runing this on a Raspberry Pi 2B+ with the latest up2date Raspbian Stretch Lite and latest up2date Pihole with the current latest Cloudflare argo-tunnel:

wget https://bin.equinox.io/c/VdrWdbjqyF/cloudflared-stable-linux-arm.tgz
tar -xvzf cloudflared-stable-linux-arm.tgz
cp ./cloudflared /usr/local/bin
chmod +x /usr/local/bin/cloudflared
cloudflared -v

(I also tried other argo-tunnel versions, but without any luck)

For me it does not make any difference if I use this tutorial https://docs.pi-hole.net/guides/dns-over-https/ or this one https://bendews.com/posts/implement-dns-over-https/ though.

...I also tried this next one (Cloudflare+Cloud9 DoH), just because we can ;-p https://www.reddit.com/r/pihole/comments/aoezvx/how_to_install_pihole_with_dns_over_https_cloud/ ...and also this other option/tutorial: https://scotthelme.co.uk/securing-dns-across-all-of-my-devices-with-pihole-dns-over-https-1-1-1-1/ ...You probably all ready might have guessed it, no luck either 👎

I can workaround it with this next line: sudo systemctl restart cloudflared 👍 If I check the status with: sudo systemctl status cloudflared everything is fine/okay, but I need to do this every time after I reboot something, it does look like a bug in my opinion. (or isn't it a bug, but an undocumented feature ;-p lol)

Reporting the same problem: when my internet connection drops, I get the expected messages in logs (running on Raspberry Pi 3):

Mar 12 09:29:16 raspberrypi cloudflared[488]: time="2019-03-12T09:29:16Z" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/.well-known/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 12 09:29:16 raspberrypi cloudflared[488]: time="2019-03-12T09:29:16Z" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/.well-known/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 12 09:29:16 raspberrypi cloudflared[488]: time="2019-03-12T09:29:16Z" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/.well-known/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
...

My first guess is that cloudflared might try to re-use existing connections, with a timeout between re-establishing the connection. In that case it would make a difference how LONG the internet connection was down (in my case, more than 2 hours) - a short disruption might be handled by TCP error correction.

BUT after the connection is restored it still doesn't return DNS responses. I can fix it by pinging 1.0.0.1 (not restarting cloudflared) . This makes me suspect a lower layer issue - routing, perhaps?