cloudflared
cloudflared copied to clipboard
cloudflare
Cloudflared hangs after internet connection severed
I am running cloudflared on ubuntu arm (raspberry pi) as a proxy dns. Whenever i restarted my router, cloudflared hangs and stops working and cant connect until i restarted the cloudflared process. Here is my cloudflared command:
cloudflared proxy-dns --port 5053 --upstream https://94.140.14.14/dns-query
Any idea of how to make cloudflared process restarts automatically if connection is lost and restored? Sorry if this has been discussed before. Thanks.
First of all, make sure you do not have version 2020.12.0 (which has issues with reconnecting tunnel connections specifically)
Thanks @vidarak. I updated to this version (https://github.com/cloudflare/cloudflared/releases/tag/2021.2.1) but the issue still persists. Here is what -version prints:
cloudflared version DEV (built unknown)
Here is the cloudflared log says after i intentionally interrupt the connection (by restarting my modem router):
Feb 07 07:00:47 home-server cloudflared[191786]: 2021-02-07T00:00:47Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Feb 07 07:00:47 home-server cloudflared[191786]: 2021-02-07T00:00:47Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 07 07:00:48 home-server cloudflared[191786]: 2021-02-07T00:00:48Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 07 07:00:48 home-server cloudflared[191786]: 2021-02-07T00:00:48Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Feb 07 07:00:49 home-server cloudflared[191786]: 2021-02-07T00:00:49Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Feb 07 07:00:50 home-server cloudflared[191786]: 2021-02-07T00:00:50Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": context deadline exceeded"
Feb 07 07:00:50 home-server cloudflared[191786]: 2021-02-07T00:00:50Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 07 07:00:51 home-server cloudflared[191786]: 2021-02-07T00:00:51Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 07 07:00:51 home-server cloudflared[191786]: 2021-02-07T00:00:51Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 07 07:00:51 home-server cloudflared[191786]: 2021-02-07T00:00:51Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
This is dig command i tried after connection interruption:
; <<>> DiG 9.16.1-Ubuntu <<>> @127.0.0.53 reddit.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
And this is the pihole log when this happens (as pihole uses cloudflared as an upstream dns):
Feb 7 07:09:31: query[A] dcg.microsoft.com from 192.168.20.100
Feb 7 07:09:31: forwarded dcg.microsoft.com to 127.0.0.1
Feb 7 07:09:31: forwarded dcg.microsoft.com to 127.0.0.1
Feb 7 07:09:32: query[A] dcg.microsoft.com from 192.168.20.100
Feb 7 07:09:32: forwarded dcg.microsoft.com to 127.0.0.1
Feb 7 07:09:32: forwarded dcg.microsoft.com to 127.0.0.1
Feb 7 07:09:33: query[A] edge.activity.windows.com from 192.168.20.100
Feb 7 07:09:33: forwarded edge.activity.windows.com to 127.0.0.1
Feb 7 07:09:33: forwarded edge.activity.windows.com to 127.0.0.1
Feb 7 07:09:33: query[A] dcg.microsoft.com from 192.168.20.100
Feb 7 07:09:33: forwarded dcg.microsoft.com to 127.0.0.1
Feb 7 07:09:33: forwarded dcg.microsoft.com to 127.0.0.1
Feb 7 07:09:35: query[A] n-devs-dcipc.tplinkcloud.com from 192.168.20.21
Feb 7 07:09:35: query[A] n-devs-dcipc.tplinkcloud.com from 192.168.20.22
Feb 7 07:09:35: query[A] dcg.microsoft.com from 192.168.20.100
Feb 7 07:09:35: forwarded dcg.microsoft.com to 127.0.0.1
Feb 7 07:09:35: forwarded dcg.microsoft.com to 127.0.0.1
Feb 7 07:09:35: query[PTR] 21.20.168.192.in-addr.arpa from 127.0.0.1
Feb 7 07:09:35: forwarded 21.20.168.192.in-addr.arpa to 127.0.0.1
Feb 7 07:09:35: forwarded 21.20.168.192.in-addr.arpa to 127.0.0.1
Pretty sure it is not the pihole's fault since i restarted cloudflared and everything works as normal again.
Yeah, can confirm i've seen this issue for quite some time now. A reboot of cloudflared, which I run in docker solves it every time.
It wouldn't surprise me if this is due to the default retries value being way too low. It is supposed to be only 5 according to Cloudflare support - with exponential backoff. You seem to have 10 entries in your log file within a couple of seconds (assuming 2 entries for every failure). It doesn't seem to do exponential backoff in your case but still max out on 5 retries. Probably just another bug that it doesn't do the backoff, but it doesn't remove the need to increase retries anyway.
I set retries to at least 99 in every config because of this wrong default retry value decision by Cloudflare.
I suspect this is the same problem as https://github.com/cloudflare/cloudflared/issues/302
can you show me the sample config of that @vidarak ?
I've added these 2 lines to configs lately (but due to a now resolved bug, the retries option only work on the most recent releases... 2020.11.11 or later): retries: 99 protocol: http2
can this be translated into command line config? i dont use any config file for cloudflared @vidarak
thanks for the suggestion @vidarak but it does not seem to fix the problem for me. i tried:
cloudflared --retries 99 proxy-dns --port 5053 --upstream https://94.140.14.14/dns-query
but the problem still persists and i have to restart cloudflared service to make it work again. here is what the log says at connection interruption:
Feb 12 06:25:06 home-server cloudflared[243291]: 2021-02-11T23:25:06Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 12 06:25:08 home-server cloudflared[243291]: 2021-02-11T23:25:08Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 12 06:25:08 home-server cloudflared[243291]: 2021-02-11T23:25:08Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 12 06:25:09 home-server cloudflared[243291]: 2021-02-11T23:25:09Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 12 06:25:09 home-server cloudflared[243291]: 2021-02-11T23:25:09Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 12 06:25:10 home-server cloudflared[243291]: 2021-02-11T23:25:10Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 12 06:25:10 home-server cloudflared[243291]: 2021-02-11T23:25:10Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Feb 12 06:25:11 home-server cloudflared[243291]: 2021-02-11T23:25:11Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 12 06:25:11 home-server cloudflared[243291]: 2021-02-11T23:25:11Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Feb 12 06:25:11 home-server cloudflared[243291]: 2021-02-11T23:25:11Z ERR failed to connect to an HTTPS backend "https://94.140.14.14/dns-query" error="failed to perform an HTTPS request: Post \"https://94.140.14.14/dns-query\": context deadline exceeded"
I have the same issue.
Raspberry Pi 3B+ Raspian Pi-Hole
$ cat /etc/debian_version
10.8
$ pihole -v
Pi-hole version is v5.2.4 (Latest: v5.2.4)
AdminLTE version is v5.4 (Latest: v5.4)
FTL version is v5.7 (Latest: v5.7)
$ cat /etc/cloudflared/config.yml
proxy-dns: true
proxy-dns-port: 5053
proxy-dns-upstream:
- https://1.1.1.3/dns-query
- https://1.0.0.3/dns-query
#Uncomment following if you want to also want to use IPv6 for external DOH lookups
- https://[2606:4700:4700::1113]/dns-query
- https://[2606:4700:4700::1003]/dns-query
$ cloudflared -v
cloudflared version 2021.1.5 (built 2021-01-18-1215 UTC)
Here is what the issue looks like when it happens:
$ service cloudflared status
● cloudflared.service - Argo Tunnel
Loaded: loaded (/etc/systemd/system/cloudflared.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-01-19 16:16:51 CST; 1 months 14 days ago
Main PID: 19786 (cloudflared)
Tasks: 11 (limit: 2063)
CGroup: /system.slice/cloudflared.service
└─19786 /home/pi/cloudflared --config /etc/cloudflared/config.yml --no-autoupdate --origincert /etc/cloudflar
Mar 05 09:19:10 dns-2 cloudflared[19786]: 9:19AM ERR failed to connect to an HTTPS backend "https://[2606:4700:4700::100
Mar 05 09:19:10 dns-2 cloudflared[19786]: 9:19AM ERR failed to connect to an HTTPS backend "https://1.0.0.3/dns-query" e
Mar 05 09:19:10 dns-2 cloudflared[19786]: 9:19AM ERR failed to connect to an HTTPS backend "https://[2606:4700:4700::111
Mar 05 09:19:10 dns-2 cloudflared[19786]: 9:19AM ERR failed to connect to an HTTPS backend "https://[2606:4700:4700::100
Mar 05 09:19:10 dns-2 cloudflared[19786]: 9:19AM ERR failed to connect to an HTTPS backend "https://1.0.0.3/dns-query" e
Mar 05 09:19:10 dns-2 cloudflared[19786]: 9:19AM ERR failed to connect to an HTTPS backend "https://[2606:4700:4700::111
Mar 05 09:19:10 dns-2 cloudflared[19786]: 9:19AM ERR failed to connect to an HTTPS backend "https://[2606:4700:4700::100
Mar 05 09:19:10 dns-2 cloudflared[19786]: 9:19AM ERR failed to connect to an HTTPS backend "https://1.0.0.3/dns-query" e
Mar 05 09:19:10 dns-2 cloudflared[19786]: 9:19AM ERR failed to connect to an HTTPS backend "https://[2606:4700:4700::111
Mar 05 09:19:10 dns-2 cloudflared[19786]: 9:19AM ERR failed to connect to an HTTPS backend "https://[2606:4700:4700::100
Restarting the service works every time to get it going again
$ sudo service cloudflared restart
$ service cloudflared status
● cloudflared.service - Argo Tunnel
Loaded: loaded (/etc/systemd/system/cloudflared.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-03-05 09:22:41 CST; 16s ago
Main PID: 17176 (cloudflared)
Tasks: 9 (limit: 2063)
CGroup: /system.slice/cloudflared.service
└─17176 /usr/local/bin/cloudflared --config /etc/cloudflared/config.yml --no-autoupdate --origincert /etc/clo
Mar 05 09:22:41 dns-1 systemd[1]: Starting Argo Tunnel...
Mar 05 09:22:41 dns-1 cloudflared[17176]: 2021-03-05T15:22:41Z INF Version DEV
Mar 05 09:22:41 dns-1 cloudflared[17176]: 2021-03-05T15:22:41Z INF GOOS: linux, GOVersion: go1.15.7, GoArch: arm
Mar 05 09:22:41 dns-1 cloudflared[17176]: 2021-03-05T15:22:41Z INF Settings: map[config:/etc/cloudflared/config.yml no-a
Mar 05 09:22:41 dns-1 cloudflared[17176]: 2021-03-05T15:22:41Z INF Adding DNS upstream url=https://1.1.1.3/dns-query
Mar 05 09:22:41 dns-1 cloudflared[17176]: 2021-03-05T15:22:41Z INF Adding DNS upstream url=https://1.0.0.3/dns-query
Mar 05 09:22:41 dns-1 cloudflared[17176]: 2021-03-05T15:22:41Z INF Adding DNS upstream url=https://[2606:4700:4700::1113
Mar 05 09:22:41 dns-1 cloudflared[17176]: 2021-03-05T15:22:41Z INF Adding DNS upstream url=https://[2606:4700:4700::1003
Mar 05 09:22:41 dns-1 cloudflared[17176]: 2021-03-05T15:22:41Z INF Starting DNS over HTTPS proxy server address=dns://lo
Mar 05 09:22:41 dns-1 systemd[1]: Started Argo Tunnel.
The issue seems to happen approximately once per month, give or take. Also, note that I have two redundant Raspberry Pis running this configuration, and they seem to fail at about the same time when they fail, and they fail the same exact way.
I believe this is fixed in #290. Try upgrading to the latest version. Feel free to reopen if that doesn't work!
@adamchalmers unfortunately it does not fix the error. I used this now:
cloudflared proxy-dns --port 5053 --upstream https://1.1.1.1/dns-query --max-upstream-conns 5
and version: cloudflared version 2021.2.5 (built 2021-02-23-1958 UTC)
and this is the log after disconnection:
Mar 07 06:23:22 home-server cloudflared[318365]: 2021-03-06T23:23:22Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:27 home-server cloudflared[318365]: 2021-03-06T23:23:27Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:27 home-server cloudflared[318365]: 2021-03-06T23:23:27Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:30 home-server cloudflared[318365]: 2021-03-06T23:23:30Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:30 home-server cloudflared[318365]: 2021-03-06T23:23:30Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:34 home-server cloudflared[318365]: 2021-03-06T23:23:34Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:35 home-server cloudflared[318365]: 2021-03-06T23:23:35Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:36 home-server cloudflared[318365]: 2021-03-06T23:23:36Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:36 home-server cloudflared[318365]: 2021-03-06T23:23:36Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:38 home-server cloudflared[318365]: 2021-03-06T23:23:38Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:38 home-server cloudflared[318365]: 2021-03-06T23:23:38Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:38 home-server cloudflared[318365]: 2021-03-06T23:23:38Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:38 home-server cloudflared[318365]: 2021-03-06T23:23:38Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:40 home-server cloudflared[318365]: 2021-03-06T23:23:40Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:40 home-server cloudflared[318365]: 2021-03-06T23:23:40Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded"
Mar 07 06:23:41 home-server cloudflared[318365]: 2021-03-06T23:23:41Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:41 home-server cloudflared[318365]: 2021-03-06T23:23:41Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:42 home-server cloudflared[318365]: 2021-03-06T23:23:42Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:42 home-server cloudflared[318365]: 2021-03-06T23:23:42Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:42 home-server cloudflared[318365]: 2021-03-06T23:23:42Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:42 home-server cloudflared[318365]: 2021-03-06T23:23:42Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:42 home-server cloudflared[318365]: 2021-03-06T23:23:42Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:42 home-server cloudflared[318365]: 2021-03-06T23:23:42Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:43 home-server cloudflared[318365]: 2021-03-06T23:23:43Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:43 home-server cloudflared[318365]: 2021-03-06T23:23:43Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:43 home-server cloudflared[318365]: 2021-03-06T23:23:43Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:43 home-server cloudflared[318365]: 2021-03-06T23:23:43Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:43 home-server cloudflared[318365]: 2021-03-06T23:23:43Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:43 home-server cloudflared[318365]: 2021-03-06T23:23:43Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:45 home-server cloudflared[318365]: 2021-03-06T23:23:45Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:45 home-server cloudflared[318365]: 2021-03-06T23:23:45Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:45 home-server cloudflared[318365]: 2021-03-06T23:23:45Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:45 home-server cloudflared[318365]: 2021-03-06T23:23:45Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:45 home-server cloudflared[318365]: 2021-03-06T23:23:45Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:46 home-server cloudflared[318365]: 2021-03-06T23:23:46Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:47 home-server cloudflared[318365]: 2021-03-06T23:23:47Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:48 home-server cloudflared[318365]: 2021-03-06T23:23:48Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:48 home-server cloudflared[318365]: 2021-03-06T23:23:48Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:49 home-server cloudflared[318365]: 2021-03-06T23:23:49Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:49 home-server cloudflared[318365]: 2021-03-06T23:23:49Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:50 home-server cloudflared[318365]: 2021-03-06T23:23:50Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Mar 07 06:23:50 home-server cloudflared[318365]: 2021-03-06T23:23:50Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
the (temporary) fix is still the same, needs to restart cloudflared manually.
Issue also exist on latest version
$ cloudflared -v
cloudflared version 2021.3.6 (built 2021-03-31-0851 UTC)
UPDATE: also exist on latest version
Systemd output after Router reconnect to WAN
Apr 03 03:42:13 raspberrypi cloudflared[25660]: 2021-04-03T01:42:13Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request
Apr 03 03:42:14 raspberrypi cloudflared[25660]: 2021-04-03T01:42:14Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request
Apr 03 03:42:14 raspberrypi cloudflared[25660]: 2021-04-03T01:42:14Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request
Apr 03 03:42:14 raspberrypi cloudflared[25660]: 2021-04-03T01:42:14Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request
Apr 03 03:42:14 raspberrypi cloudflared[25660]: 2021-04-03T01:42:14Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request
Apr 03 03:42:14 raspberrypi cloudflared[25660]: 2021-04-03T01:42:14Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request
Apr 03 03:42:14 raspberrypi cloudflared[25660]: 2021-04-03T01:42:14Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request
Apr 03 03:42:14 raspberrypi cloudflared[25660]: 2021-04-03T01:42:14Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request
Apr 03 03:42:14 raspberrypi cloudflared[25660]: 2021-04-03T01:42:14Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request
Apr 03 03:42:15 raspberrypi cloudflared[25660]: 2021-04-03T01:42:15Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request
Also exists on 2021.4.0. Any loss of, or failover of, the WAN and Cloudflared just hangs until manually restarted....
Apr 19 14:05:51 PiHole cloudflared[1306]: 2021-04-19T13:05:51Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Apr 19 14:05:56 PiHole cloudflared[1306]: 2021-04-19T13:05:56Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Apr 19 14:06:42 PiHole cloudflared[1306]: 2021-04-19T13:06:42Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Apr 19 14:06:47 PiHole cloudflared[1306]: 2021-04-19T13:06:47Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Apr 19 14:08:24 PiHole cloudflared[1306]: 2021-04-19T13:08:24Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Apr 19 14:08:29 PiHole cloudflared[1306]: 2021-04-19T13:08:29Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
This makes Cloudflared completely unusable for me tbh.
Having the same problem, which for me makes cloudflared unusable for me as well. Switching to standard DNS until this can be resolved.
Having the same problem, which for me makes cloudflared unusable for me as well. Switching to standard DNS until this can be resolved.
I've went with installing Unbound and recursive resolving using the root servers.
Having same problem with Unifi and backup LTE connection, if it switches to backup connection and a different external IP then cloudflared hangs. Running latest editions of Pi-Hole, Cloudflared and Raspberry PI OS.
I have this issue too. Except mine happens within 5mins of starting cloudflared service .
pi@pihole:~ $ pihole -v
Pi-hole version is v5.7 (Latest: v5.8.1)
AdminLTE version is v5.9 (Latest: v5.10.1)
FTL version is v5.12.1 (Latest: v5.13)
pi@pihole:~ $ cloudflared -v
cloudflared version 2022.1.0 (built 2022-01-06-1626 UTC)
pi@pihole:~ $
oudflared/config.yml --no-autoupdate --origincert /etc/cloudfla>
ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns->
ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns->
ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns->
ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns->
ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns->
ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns->
ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns->
ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns->
ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns->
ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns->
~
abled; vendor preset: enabled)
9min ago
EDIT: Changing to upstream--conns 50 has worked for me as well.
Having same problem with Unifi and backup LTE connection, if it switches to backup connection and a different external IP then cloudflared hangs. Running latest editions of Pi-Hole, Cloudflared and Raspberry PI OS.
I am experiencing the same problem with cloudflared tunnel connections. If network is unavailable for a short amount of time the cloudflared service stucks after the defined --retries (default 5) value.
As the time between retries increases exponentially it isn't really helpful to increase retries. Wouldn't it be possible to add an option to define the wait-between-retries time to allow the network to reconnect again or to kill the service so it can be brought up again by e.g. systemd or Docker?
Having same problem with Unifi and backup LTE connection, if it switches to backup connection and a different external IP then cloudflared hangs. Running latest editions of Pi-Hole, Cloudflared and Raspberry PI OS.
I am experiencing the same problem with
cloudflared tunnelconnections. If network is unavailable for a short amount of time the cloudflared service stucks after the defined--retries(default 5) value. As the time between retries increases exponentially it isn't really helpful to increase retries. Wouldn't it be possible to add an option to define the wait-between-retries time to allow the network to reconnect again or to kill the service so it can be brought up again by e.g. systemd or Docker?
Better idea - use dnscrypt. Problem solved.
No one at CF cares about this issue that's been ongoing for years now. It won't be fixed anytime soon.
https://github.com/DNSCrypt/dnscrypt-proxy
Now I can rest assured that momentary internet drops won't end up in much longer outages.
I too ran into this issue with cloudflared tunnel when running a BlueBubbles Server on macOS and intermittent internet issues.
I have the following launch agent set up as a workaround and it seems to work okay. It basically exits the process if grep finds: "INF Retrying connection in up to 16s seconds" in the output. Then the launch agent will auto-restart thanks to the KeepAlive flag.
/Users/myusername/Library/LaunchAgents/com.cloudflare.cloudflared.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.cloudflare.cloudflared</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/bin/bluebubbles-tunnel.sh</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>StandardOutPath</key>
<string>/Users/myusername/Library/Logs/com.cloudflare.cloudflared.out.log</string>
<key>StandardErrorPath</key>
<string>/Users/myusername/Library/Logs/com.cloudflare.cloudflared.err.log</string>
<key>KeepAlive</key>
<dict>
<key>SuccessfulExit</key>
<false/>
</dict>
<key>ThrottleInterval</key>
<integer>5</integer>
</dict>
</plist>
/usr/local/bin/bluebubbles-tunnel.sh
#! /bin/bash
echo "" > /Users/myusername/Library/Logs/com.cloudflare.cloudflared.out.log
echo "" > /Users/myusername/Library/Logs/com.cloudflare.cloudflared.err.log
/usr/local/bin/cloudflared tunnel run 2>&1 | while read line; do
echo "$line"
echo "$line" | grep "INF Retrying connection in up to 16s seconds" -q && { exit 1; break; }
done
I am having the same issue and it's disrupting the service I provide. It would be great if this could be fixed soon. I'll try and find a workaround, but it won't be possible until I can get physical access to the server.
The tunnel side of the issue is https://github.com/cloudflare/cloudflared/issues/724
There’s a commit in there that fixes this problem. You can build this from source if the need is urgent.
note that this is only for tunnel but not dns-proxy.
FWIW, My work-around is to force cloudflared to restart every 10 minutes or so.
$ cat /etc/cron.d/cloudflared
# /etc/cron.d/cloudflared: crontab entries for the cloudflared package
#
#
# Important Note! This cronjob will NOT be executed if you are
# running systemd as your init system. If you are running systemd,
# the cronjob.timer function takes precedence over this cronjob. For
# more details, see the systemd.timer manpage.
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
1 * * * * root test -x /usr/local/bin/cloudflared && systemctl restart cloudflared
10 * * * * root test -x /usr/local/bin/cloudflared && systemctl restart cloudflared
20 * * * * root test -x /usr/local/bin/cloudflared && systemctl restart cloudflared
30 * * * * root test -x /usr/local/bin/cloudflared && systemctl restart cloudflared
40 * * * * root test -x /usr/local/bin/cloudflared && systemctl restart cloudflared
50 * * * * root test -x /usr/local/bin/cloudflared && systemctl restart cloudflared
Also having this issue, My instance only ever seems to last for about a day before erroring out and then I need to remote onto the server an close the current instance and start a new one. I have a backup tunnel luckily so its not causing any downtime for me but its a huge pain in the ass because this is my main server and I would like my traffic to go via this server ideally. If anyones got any solution for windows, please let me know. I am trying to find a way to terminate the script automatically at a certain time or maybe upon the error message and start a new one.
