cloudflare issues: multiple attempts/black screen

[billettg]

Describe the bug Clicking on the "Connect" button under "Desktop" or "Terminal" results in "Disconnected" approximately 9/10 times. Other times it will connect successfully. The disconnection is immediately shown after clicking "Connect".

To Reproduce Steps to reproduce the behavior:

1. Go to "Terminal"
2. Click on "Connect"
3. See "Disconnected"

Expected behavior Connects successfully on every attempt.

Screenshots If applicable, add screenshots to help explain your problem.

Server Software (please complete the following information):

• OS: Ubuntu 22.04.3 LTS
• Virtualization: NA
• Network: WAN/CloudFlare DNS + proxy (not tunnel)
• Version: 1.1.10
• Node: 12.22.9

Client Device (please complete the following information):

• Device: 3 unique devices
• OS: macOS, Linux, Windows
• Network: Remote over WAN
• Browser: Google Chrome/Safari
• MeshCentralRouter Version: NA

Additional context The problem seems to only occur using CloudFlare, so I think that the proxy is causing websocket disconnection. The MeshCentral VM is hosted on the Hetzner platform. Others face the same issue (e.g https://www.reddit.com/r/MeshCentral/comments/15y28x3/random_disconnects_behind_cloudflare/)

Your config.json file

{
  "settings": {
    "cert": "mesh.example.com",
    "wanonly": true,
    "port": 443,
    "aliasport": 443,
    "redirport": 80,
    "rediraliasPort": 80,
    "webrtc": true,
    "wscompression": true,
    "allowlogintoken": true,
    "trustedproxy": "CloudFlare"
  },
  "domains": {
    "": {
      "newaccounts": false,
      "usernameisemail": true,
      "certurl": "https://mesh.example.com"
    }
  }
}

[vesector]

+1

[si458]

it could be webrtc issue? have you tried with webrtc set to false?

[si458]

also nodejs 12 is now EOL, please update node to the latest LTS (18)

[vesector]

I have tested with WebRTC disabled and also enabled, having the same behaviour.

One thing to note is that this issue started a few weeks ago.

[billettg]

Updated nodejs to 18.17.1 and same issue with or without webrtc enabled. I rebooted the server and checked nodejs version with the --version parameter.

The logs show connected then immediately disconnected. When it works I noticed that "Relay holding: * (::1) Authenticated" shows as well.

WEBREQUEST: (-) /meshrelay.ashx/.websocket?p=1&nodeid=node//iWMDZtF%24QteFec1amvHRz7nA8c4SqcXNrz7d7HN5JYxOTNwrOvK%24WOqkZ9XJjHuq&id=ma8286xee29&rauth=VEcPv1E75FdSda%24JPWUN7XeUM2plL7mDexVJ37kpZ%24v4selPtY3Kb%24Voz%40maz2MGG1E3IP4Pm9VcASvz3YSGyGo9CNWntAzVx3A%24BLwfSdTIPN4by%40o7 COOKIE: Decoded AESGCM cookie: {"ruserid":"user//gareth","x":"BQF0kfBA","time":1692713115000,"dtime":6268} RELAY: Relay connected: ma8286xee29 (- -> -) DISPATCH: DispatchEvent [ '', 'user//gareth' ] RELAY: Relay disconnect: ma8286xee29 (8- --> -) DISPATCH: DispatchEvent [ '', 'user//gareth', 'node//iWMDZtF$QteFec1amvHRz7nA8c4SqcXNrz7d7HN5JYxOTNwrOvK$WOqkZ9XJjHuq', 'mesh//DvVA69QdUY7bH8nbnnI4X@cSpkJOl$iIExyDvvBtFEudOUwUuEn5C2QNrHepdL6A' ]

[jwiener3]

+1 I see the same issues, immediate disconnect for different hosts at different locations. All the same symptoms as stated above. I am proxying through Cloudflare, I am not sure if others are doing the same.

[hiddenpcmaster]

+1

[dooley74]

Same here, just started a few days ago though. Running through Cloudflare as well.

[jirijanata]

+1 Also running through Cloudflare. Actually the problem first occured for me after Upgrade from 1.1.8 to 1.1.10.

[supra36]

+1 same, although it didn't start immediately after 1.1.8=>1.1.10 transition, it started giving issues gradually and now it affects all agents no matter the OS.

Real-time monitoring using graphs works fine, however it fails to establish any other type of connection, let it be webrtc, stream, rdp, terminal or file transfer. MeshCentral Router also fails to RDP in "Configuring remote session" phase.

Also, I'm running mesh through cloudflare too.

[si458]

this sounds to me like a cloudflare issue not a meshcentral issue, as nothing has changed recently to do with proxies or tunneling can anybody donate a domain/cloudflare details and i could look at it in my freetime?

[supra36]

I took a look and filtered websocket related issues in google search from last 7 days. These are results.

https://community.cloudflare.com/t/websocket-not-stable-some-time-connect-some-time-no/547638

https://community.cloudflare.com/t/websocket-problem/547094

https://community.cloudflare.com/t/protected-web-socket-connection-dies-after-30-secs-cloudflare-ipv6-issue/547127

Unfortunately these haven't been answered.

Also it seems that CloudFlare bottlenecks websocket connections after reaching certain level of connections (or spikes). Does anybody know if CloudFlare started limiting websocket connections just recently or changed anything related to these limits or to how these are handled?

https://developers.cloudflare.com/support/network/using-cloudflare-with-websockets/

[vesector]

Great find @supra36, I have the feeling it is Cloudflare related more than the actual update of MC to .10... I am not 100% though.

I did a tcpdump and I saw Cloudflare giving [F] and [R] flags while reproducing the issue.

Would be nice if someone from the community that experiences the same issue and has a Cloudflare business or higher tier subscription opens a ticket with their support to see if this gets an answer from their side... as using the free tier there is no chance of support afaik.

[NiceGuyIT]

There's a Cloudflare Developers on Discord. Might be useful to ask around.

A post 3 days ago mentioned websocket disconnects when using IPv6 but not when using IPv4. Someone mentioned disabling IPv6 in Network -> IPv6 Compatibility.

[vesector]

Hi @NiceGuyIT,

Thank you for the suggestion.

I have disabled IPv6 through the API and the issue persists.

For anyone that wants to give it a try... I used PowerShell:

$headers=@{} $headers.Add("Content-Type", "application/json") $headers.Add("X-Auth-Email", "YOURCFEMAIL") $headers.Add("X-Auth-Key", "YOURAPIKEY") $response = Invoke-WebRequest -Uri 'https://api.cloudflare.com/client/v4/zones/YOURZONEID/settings/ipv6' -Method PATCH -Headers $headers -ContentType 'application/json' -Body '{ "value": "off" }'

[supra36]

Just to let you know, I ditched cloudflare and used let's encrypt. It's now working fine.

[billettg]

Just to let you know, I ditched cloudflare and used let's encrypt. It's now working fine.

I decided to provision another Ubuntu server and installed MC 1.1.10 with Let's Encrypt certificate rather than CloudFlare and it also works fine. I wonder what changes have been made on CloudFlare side to be dropping the websocket connections, it must be something recent. I'm sure their support would be able to comment but unfortunately we are on a free tier without any technical support.

[vesector]

As per the questioning raised by @frogweh on #5309 he seems to be using Nginx and not CloudFlare and having the same issue... would be good to test that to see if it's reverse proxy in general or if we can completely isolated to CF. Also, would be good to try with an older version of .10 so we can rule out the update of MC completely. Will try to do this later this week.

[si458]

@iribarrenjg if you read the post #5309 he says he uses a cloudflare certificate and if you read the logs shows heads for cf connecting from, so he is indeed using cloudflare! i do think the is an issue with cloudflare?

[mon5termatt]

Having the same issue, also using cloudflare.

[Vista2003]

+1 on 1.1.10 (Raspberry Pi OS 64 Bit via Cloudflare and Nginx Proxy Manager)

[jwiener3]

Like others here, I changed my DNS records away from Cloudflare's proxy and things started working as expected. Also, this was happening before 1.1.10. I upgraded to 1.1.10 in hopes of it being a MeshCentral issue. Now I am trying to figure out how to lock down my environment without Cloudflare's rules. I know it is possible, but Cloudflare made it easy :).

[si458]

@jwiener3 you can do an IP allow list for agents and clients which might help? https://github.com/Ylianst/MeshCentral/blob/b1d2d1aea96c5de48be210e89961347cb6b8b72b/meshcentral-config-schema.json#L592-L628

[jwiener3]

@jwiener3 you can do an IP allow list for agents and clients which might help?

Thanks, I will take a look at that.

[mon5termatt]

So I can just disable the proxy checkmark on cloudflare? I have my domain registered with them.

[mon5termatt]

[jwiener3]

So I can just disable the proxy checkmark on cloudflare? I have my domain registered with them.

Yes that is what I did, and then I had to open up my ACL on the server that was hosting Meshcentral to allow connections from anywhere, as I only had it allowing connections from cloudflare IP space.

[appleimperio]

+1 docker Raspberry pi with Cloudflare tunnels. Today I update the Cloudflare tunnel app and now is not connecting at all

[jirijanata]

Since yesterday I'm not able to connect anymore. A few days before were a few tries were needed, but now it doesn't work at all.

[themanbornwithin]

+1 for nginx reverse proxy and utilizing Cloudflare Proxy