Cloudflared doesn't recover itself when network is unavailable

[ferferga]

Hello

I'm using cloudflared proxy-dns --port 5053 --max-upstream-conns 0. When my ISP connection goes down, cloudflared prompty logs the following:

cloudflared[3095]: 2021-09-03T15:21:19Z ERR failed to connect to an HTTPS backend "https://1.0.0.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.0.0.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

However, once the connection recovers, it keeps spamming the same log message over and over, even when the connection has been recovered successfully (i.e I can do pings to 1.1.1.1 without issues). I need to restart the service I configured for it (kill the process and start it again) so it starts answering queries again.

[sudarshan-reddy]

Can you try without the ‘max-upstream-conns’ set to a non zero value?

[xPaw]

Same here.

Sep 10 20:11:38 raspberrypi cloudflared[31063]: 2021-09-10T17:11:38Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:11:38 raspberrypi cloudflared[31063]: 2021-09-10T17:11:38Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:11:38 raspberrypi cloudflared[31063]: 2021-09-10T17:11:38Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded"
Sep 10 20:11:43 raspberrypi cloudflared[31063]: 2021-09-10T17:11:43Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:11:43 raspberrypi cloudflared[31063]: 2021-09-10T17:11:43Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:11:47 raspberrypi cloudflared[31063]: 2021-09-10T17:11:47Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:11:47 raspberrypi cloudflared[31063]: 2021-09-10T17:11:47Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:11:48 raspberrypi cloudflared[31063]: 2021-09-10T17:11:48Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:11:48 raspberrypi cloudflared[31063]: 2021-09-10T17:11:48Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:11:52 raspberrypi cloudflared[31063]: 2021-09-10T17:11:52Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:11:52 raspberrypi cloudflared[31063]: 2021-09-10T17:11:52Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

I don't specify max-upstream-conns.

/usr/local/bin/cloudflared proxy-dns --port 5053 --upstream https://1.1.1.1/dns-query --upstream https://1.0.0.1/dns-query

I have to kill the process because just doing restart isn't enough.

[ferferga]

@sudarshan-reddy I've used Cloudflared in the past without that argument and I also had issues. They got fixed at some point but seems that they're back now.

I'm testing without the parameter now, just to make sure, but I can't interrupt the Internet connection until next week to test, as people in home rely on it. Anyway, the problem is easily reproducible, just take down Internet and see, there's no special way to reproduce like waiting X seconds, overflow it with requests, etc.

Also, probably important to mention that, as @xPaw , I'm using a Pi 4, so maybe it's an armhf-only issue.

[walksonair]

Just experienced this myself this week. Multiple power failures caused no internet on local network. I had to manually restart Pihole DNS resolver to reestablish internet.

There may be something going on w PiHole in this mix: when I unplug the ethernet from my RPi, wait a moment and then plug it back in, DNS service isn't able to get back to normal with out asking PiHole to restart the DNS resolver. I've sent this issue over to Pihole admins as well.

Further troubleshooting: Unplugging the ethernet cord, waiting and re-plugging it back in when I selected a different DNS, 'Cloudfare', instead of 127.0.0.1#5053, and ensuring that I turned off 'Use DNSSEC', results in the same behavior w/ no improvement. It is definitely strange for me on what is going on.

Please advise on any further debugging/help needed. Thanks for looking into this.

[ivanjx]

https://github.com/cloudflare/cloudflared/issues/306

[marcosvrs]

Same here.

[sudarshan-reddy]

Just to confirm, are you using cloudflared in proxy-dns mode?