cloudflared icon indicating copy to clipboard operation
cloudflared copied to clipboard

🐛 Issues egressing to Cloudflare with quic protocol

Open zzduci opened this issue 2 years ago • 17 comments

zzduci avatar Mar 18 '23 15:03 zzduci

is anyone here?please help

zzduci avatar Mar 19 '23 07:03 zzduci

cloudflared tunnel --protocol http2 --no-autoupdate run --token ,this command can change the tunnel status to healthy,but it cat work well .after a few time ,the tunnel status is down

zzduci avatar Mar 19 '23 07:03 zzduci

cloudflared tunnel --protocol http2 --no-autoupdate run --token yourtokenhere ,this command can change the tunnel status to healthy,but it cat work well .after a few time ,the tunnel status is down

zzduci avatar Mar 19 '23 07:03 zzduci

now it is down status image

zzduci avatar Mar 19 '23 07:03 zzduci

I am new to tunnels, but recently have seen similar errors. Have a look at #917, I think these might be related. As you describe and many described it on the #917, your tunnel disconnects from Cloudflare's edge (degrades) servers and does it once every minute or so, reconnecting once you restart the cloudflared itself. If it is right, there is currently no fix except for switching/forcing the QUICK protocol, more on #917.

alexandrutocar avatar Mar 20 '23 23:03 alexandrutocar

I am new to tunnels, but recently have seen similar errors. Have a look at #917, I think these might be related. As you describe and many described it on the #917, your tunnel disconnects from Cloudflare's edge (degrades) servers and does it once every minute or so, reconnecting once you restart the cloudflared itself. If it is right, there is currently no fix except for switching/forcing the QUICK protocol, more on #917.

thanks for your help ,i mean can i wait for Cloudflared to fix it?

zzduci avatar Mar 21 '23 01:03 zzduci

I have a feeling that there is something more to these failures, maybe some kind of obscure issue with the edge servers. So either this, and we get to see an update on cloudflare's blog and a fix. Or there will come some instructions on how to fix stuff related to networking specific issues. It is being investigated right now, and I believe there will be more updates on the mentioned issue.

alexandrutocar avatar Mar 21 '23 11:03 alexandrutocar

I have a feeling that there is something more to these failures, maybe some kind of obscure issue with the edge servers. So either this, and we get to see an update on cloudflare's blog and a fix. Or there will come some instructions on how to fix stuff related to networking specific issues. It is being investigated right now, and I believe there will be more updates on the mentioned issue.

cloudflared tunnel --protocol http2 --no-autoupdate run --token yourtokenhere ,this command can fix it ,but not useful enough

zzduci avatar Mar 21 '23 12:03 zzduci

Based off of your log dumps, I think you are having trouble running a tunnel with the QUIC protocol. Additionally, it looks like you were able to find and use the --protocol http2 as a backup and it seemed to solve your connection issues.

It's true, we do find some customers who have issues egressing UDP traffic to reach Cloudflare, it's a really complicated and difficult process to remediate who in the route is dropping these UDP packets: your computer's firewall, router, ISP, etc.

DevinCarr avatar Mar 21 '23 21:03 DevinCarr

Were you able to validate that in the cloudflared logs that the tunnel was still connected? By running with --loglevel debug you can provide more detailed information about the status of the tunnel during startup and, if it's failing consistently to connect to the edge, you should see information in the logs to state that.

DevinCarr avatar Mar 23 '23 17:03 DevinCarr

Were you able to validate that in the cloudflared logs that the tunnel was still connected? By running with --loglevel debug you can provide more detailed information about the status of the tunnel during startup and, if it's failing consistently to connect to the edge, you should see information in the logs to state that.

would you tell the full command? is cloudflared tunnel run tunnelname --loglevel debug?

zzduci avatar Mar 24 '23 12:03 zzduci

cloudflared tunnel --loglevel debug run <tunnelname>

DevinCarr avatar Mar 24 '23 17:03 DevinCarr

After testing, both http2 and h2mux can establish a connection normally, only quic cannot establish a connection. I think it should be caused by this problem, I don't know if it is right

zzduci avatar Mar 25 '23 03:03 zzduci

I get the exact same errors. but from a location in Europe.

alexandrutocar avatar Mar 25 '23 08:03 alexandrutocar

I get the exact same errors. but from a location in Europe. follow commands: cloudflared tunnel login

cloudflared tunnel create [tunnelname]

cloudflared tunnel route dns [tunnelname] [domain]

vi config.yml ###config.yml:

tunnel: alist credentials-file: /root/.cloudflared/ab49d2b8-057d-4ee9-9675-c9dca34e85e4.json protocol: http2 originRequest: connectTimeout: 30s noTLSVerify: false

ingress:

  • hostname: XXXXXXX.com service: http://localhost:[yourport]
  • service: http_status:404

nano /etc/systemd/system/cloudflared.service

#cloudflared.service:

[Unit] Description=cloudflared After=network.target

[Service] TimeoutStartSec=0 Type=notify ExecStart=/usr/bin/cloudflared --loglevel debug --transport-loglevel warn --config /root/.cloudflared/config.yml tunnel run [name] Restart=always RestartSec=5s

[Install] WantedBy=multi-user.target

reboot

now it is working ,i think the most important is " protocol: http2"

zzduci avatar Mar 25 '23 12:03 zzduci

Running cloudflared tunnel --url localhost:8080 --http2-origin with version 2023.7.3, after almost 24h of uptime, I got this:

2023-09-26T13:28:42Z INF Unregistered tunnel connection connIndex=0 event=0 ip=198.41.200.43
2023-09-26T13:28:42Z ERR Failed to serve quic connection error="timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.43
2023-09-26T13:28:42Z ERR Serve tunnel error error="timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.43
2023-09-26T13:28:42Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=198.41.200.43
2023-09-26T13:28:48Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.193
2023-09-26T13:28:48Z INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.200.193
2023-09-26T13:28:55Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.107
2023-09-26T13:28:55Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.192.107
2023-09-26T13:29:08Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63
2023-09-26T13:29:08Z INF Retrying connection in up to 16s connIndex=0 event=0 ip=198.41.200.63
2023-09-26T13:29:15Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.33
2023-09-26T13:29:15Z INF Retrying connection in up to 32s connIndex=0 event=0 ip=198.41.200.33
2023-09-26T13:29:44Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.27
2023-09-26T13:29:44Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.192.27
2023-09-26T13:30:20Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.37
2023-09-26T13:30:20Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.192.37
2023-09-26T13:30:30Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.53
2023-09-26T13:30:30Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.53
2023-09-26T13:30:55Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.193
2023-09-26T13:30:55Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.193
2023-09-26T13:31:15Z INF Registered tunnel connection connIndex=0 connection=35171487-fc83-47a0-85a2-755f1985b644 event=0 ip=198.41.192.167 location=lis01 protocol=quic
2023-09-26T14:47:38Z ERR update check failed error="no release found"

ruimarinho avatar Sep 26 '23 15:09 ruimarinho