Cloudflare Tunnel randomly stops serving traffic

[Albonycal]

Cloudflare tunnel randomly not working The daemon is still running but no traffic reaches it.. restarting the daemon fixes it for a while, but then it happens again. No pattern observed, just randomly stops working

Environment and versions

• OS: Ubuntu Server 20.04
• Architecture: AMD64
• Version: 2022.7.1
• Running as a systemd service

Logs and errors

Jul 17 12:08:33 phoenix cloudflared[470354]: 2022-07-17T06:38:33Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:33 phoenix cloudflared[470354]: 2022-07-17T06:38:33Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:34 phoenix cloudflared[470354]: 2022-07-17T06:38:34Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:34 phoenix cloudflared[470354]: 2022-07-17T06:38:34Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:34 phoenix cloudflared[470354]: 2022-07-17T06:38:34Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:34 phoenix cloudflared[470354]: 2022-07-17T06:38:34Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:35 phoenix cloudflared[470354]: 2022-07-17T06:38:35Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:35 phoenix cloudflared[470354]: 2022-07-17T06:38:35Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:36 phoenix cloudflared[470354]: 2022-07-17T06:38:36Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:36 phoenix cloudflared[470354]: 2022-07-17T06:38:36Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:36 phoenix cloudflared[470354]: 2022-07-17T06:38:36Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:36 phoenix cloudflared[470354]: 2022-07-17T06:38:36Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>
Jul 17 12:08:36 phoenix cloudflared[470354]: 2022-07-17T06:38:36Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>

Its showing this error, but it used to show some TCP connect error for the URL inside LAN

Additional context I am using the --protocol http2 flag because quic doesn't work host: mirror.albony.xyz cloudflare email: [email protected]

[sudarshan-reddy]

I am using the --protocol http2 flag because quic doesn't work

Can you explain what you mean by doesn't work? We'll need a little more information to help you. :)

Incoming request ended abruptly: context canceled

This basically means the requests to your tunnel were cancelled because of a timeout. Is your origin service slow my any chance?

[Albonycal]

This basically means the requests to your tunnel were cancelled because of a timeout. Is your origin service slow my any chance?

no? when cloudflared is "down" my origin services works fine, and at gigabit LAN speeds.

[Albonycal]

Can you explain what you mean by doesn't work? We'll need a little more information to help you. :)

it fails to connect using quic, I have tried everything, even setting up port triggering for it so now I force it to use HTTP2 before cloudflared use to use HTTP2 by default idk why it tries to use quic now

[Albonycal]

This basically means the requests to your tunnel were cancelled because of a timeout. Is your origin service slow my any chance?

and even if it's slow, cloudflare tunnel shouldn't just crash ?

[Albonycal]

also, I have two instances of cloudflared now, both do the same thing.

[sudarshan-reddy]

it fails to connect using quic, I have tried everything, even setting up port triggering for it so now I force it to use HTTP2 before cloudflared use to use HTTP2 by default idk why it tries to use quic now

What is your system running on? Do you have UDP egress enabled? quic is the direction we want to take the future of cloudflared for, for both performance and stability reasons.

[sudarshan-reddy]

and even if it's slow, cloudflare tunnel shouldn't just crash ?

Can you share crash logs? I only see

Jul 17 12:08:36 phoenix cloudflared[470354]: 2022-07-17T06:38:36Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=>

[Albonycal]

uh by "crash" it doesn't really crash, the daemon is still running but it doesn't serve traffic

[Albonycal]

also, is this normal behaviour? notice the space between the bars, it drops the speed when I am downloading something and then regains it

[sudarshan-reddy]

That is expected behaviour since your bandwidth is shared.

[sudarshan-reddy]

it fails to connect using quic, I have tried everything, even setting up port triggering for it

Did you enable UDP egress in your firewall?

[Albonycal]

yea I did

On Wed, Jul 20, 2022 at 3:55 PM Sudarshan Reddy @.***> wrote:

it fails to connect using quic, I have tried everything, even setting up port triggering for it

Did you enable UDP egress in your firewall?

— Reply to this email directly, view it on GitHub https://github.com/cloudflare/cloudflared/issues/698#issuecomment-1190101986, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP7TNJ7FXWWHKPJLHPQ6LMDVU7H2TANCNFSM53ZPRVSQ . You are receiving this because you authored the thread.Message ID: @.***>

[Albonycal]

I added this to my router config

[Albonycal]

tried and QUIC works, getting some errors on the console but it seems to work, so ig it's fine?

update: yea quic seems to work

[sudarshan-reddy]

Nice! Are you still having these issues?

[Albonycal]

hmm no.. but these are still there and I see some quic errors

[Albonycal]

still getting a lot of "incoming request ended abruptly"

[sudarshan-reddy]

This is indicative of your clients closing these requests. What is the experience you get on the eyeball side?

[Albonycal]

This is indicative of your clients closing these requests.

That makes sense

what do you mean by the "eyeball" side?

[Albonycal]

That particular service is being monitored here, https://archlinux.org/mirrors/albony.xyz/ and I see some errors which shouldn't be there..

[Albonycal]

i rechecked and confirmed that the same issue is happening.. and restarting the daemon "fixes" it temperorrily,,, but it happens again

[Albonycal]

I wonder if a burst of high traffic can trigger a bug in cloudflared or something that makes it unresponsive.. This machine is quite old so maybe that's the issue? but still it shouldn't act like this, looks like a cloudflared bug to me ?

[sudarshan-reddy]

@joliveirinha may have found a possible solution.

[Albonycal]

:eyes: :crossed_fingers:

[Albonycal]

Hey! has there been any progress on this issue? anything I can help with? Thank you

[joliveirinha]

Hi @Albonycal ,

We did some improvements, but looking at this issue, I don't know if they are related to the particular one you mention here.

Looking at the logs, as @sudarshan-reddy mentioned, it just seems that eyeball left before the request finished. We are doing a release next week, let's see if any of the fixes actually helped here.

[don4of4]

I am using the --protocol http2 flag because quic doesn't work

Can you explain what you mean by doesn't work? We'll need a little more information to help you. :)

I am going to raise my hand and say we have been very frustrated by this and a litany of issues with the tunnel recently. We were seeing requests truncated (return 200), but have no body, causing P1 downstream issues since standard retry logic wasn't activated. We saw issues under 2022.4.1 and 2022.8.2

@tomhuang12 is working with support under ticket 2535568 -- but please let us know what we can do to support. We would like to see this stabilized ASAP.

[Albonycal]

If it helps, when I say it "stops serving requests" it doesn't return the tunnel error or 503, the requests gets timed out

[Albonycal]

This still happens :(

[morpig]

can confirm this still happens. running cloudflared v2022.10.3 in kubernetes setup w/ http2 protocol set.

can send some CF-rays through ticket or here if needed.