web relay not working with text/event-stream

[si458]

Describe the bug when i relay my freepbx web panel, it sits on the dashboard and then it wont load any other pages, it appears to be loading the 'live network usage' graph from a URL which is returning a datatype of 'text/event-stream' from the looks of it (googling) the event-stream is a server side stream, so the client opens the http request then listens to the replys from the server, so this appears to be blocking any other HTTP requests

https://www.w3schools.com/html/html5_serversentevents.asp - help docs

To Reproduce Steps to reproduce the behavior:

1. connect to a freepbx server on port 80 using web relay (HTTP option)
2. dashboard loads
3. graphs arent being loaded, but click another menu option, nothing happens just sits there

Expected behavior for it to relay correctly

Screenshots N/A

Server Software (please complete the following information):

• OS: Alpine
• Virtualization: Docker
• Network: Hybrid, Traefik Reverse, SSL Offload
• Version: 1.0.63
• Node: 16.16.0

Client Device (please complete the following information):

• Device: Laptop
• OS: Windows
• Network: Remote Over WAN
• Browser: [e.g. Google Chrome] Firefox 103.0.1
• MeshCentralRouter Version: [if applicable] N/A

Remote Device (please complete the following information):

• Device: Server
• OS: FreePBX Distro (Sangoma Linux 7 (Core))
• Network: Remote Over WAN
• Current Core Version (if known): Apr 4 2022, 2277534278

Additional context this doesnt happen when relaying via meshcentralrouter as that does a physical TCP relay rather than a HTTP relay

if you go into the MC panel then CLOSE the TCP relay session thats listed, the connection is closed in the webpage then suddenly it loads all other HTTP requests and menu options you clicked

Your config.json file

{
  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "__comment1__": "This is a simple configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.",
  "__comment2__": "See node_modules/meshcentral/sample-config-advanced.json for a more advanced example.",
  "settings": {
    "_cert": "myserver.mydomain.com",
    "_WANonly": true,
    "_LANonly": true,
    "_sessionKey": "MyReallySecretPassword1",
    "_port": 443,
    "_aliasPort": 443,
    "_redirPort": 80,
    "_redirAliasPort": 80
  },
  "domains": {
    "": {
      "_title": "MyServer",
      "_title2": "Servername",
      "_minify": true,
      "_newAccounts": true,
      "_userNameIsEmail": true
    }
  },
  "_letsencrypt": {
    "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.",
    "email": "[email protected]",
    "names": "myserver.mydomain.com",
    "production": false
  }
}

[Ylianst]

Yes... indeed your exactly right. I don't even have to look at the code, a "text/event-stream" HTTP request will cause MeshCentral to pile up all the other HTTP calls behind it and that will be bad. I already have code to stop that from happening for websockets since they act in exactly the same way.

[si458]

Yes... indeed your exactly right. I don't even have to look at the code, a "text/event-stream" HTTP request will cause MeshCentral to pile up all the other HTTP calls behind it and that will be bad. I already have code to stop that from happening for websockets since they act in exactly the same way.

is the anyway of fixing this then as the only way i can fix the issue currently is click a link then go into MC and force close the tcp session until the page reloads to where it needs to go

[Ylianst]

Yes, I am working on it now :) Once I am ready, I may have you copy that new "apprelay.js" from GitHub to your server for testing since I am not going to be able to replicate your exact situation.

[si458]

Yes, I am working on it now :) Once I am ready, I may have you copy that new "apprelay.js" from GitHub to your server for testing since I am not going to be able to replicate your exact situation.

brilliant! any private testing required, email, twitter, etc on my profile, you can contact me https://github.com/si458

[Ylianst]

I just checked in the fix for it. You can grab the new version of the file here. Keep your old version of "apprelays.js" just in case, but put the new on it and let me know if it works. All it does is detect "text/event-stream" requests and marks it like a websocket as a request that will never free up.

[si458]

I just checked in the fix for it. You can grab the new version of the file here. Keep your old version of "apprelays.js" just in case, but put the new on it and let me know if it works. All it does is detect "text/event-stream" requests and marks it like a websocket as a request that will never free up.

that made it completely worse, i couldnt even get it to load the page correctly and POST to a login, it just seemed to just sat there waiting for a reply from the relay at LOGIN

EDIT: ok it does eventually load the login page (takes about 30 seconds, very very long delay for some reason on very first relay, anything after is quicker) and it does POST the login but again takes about 30 seconds, then loads up the dashboard but the graph isnt showing any data and no data from the stream, but i click a menu option for another page and it just sits their still doing nothing, being blocked until i close the tcp relay in the MC web ui

[Ylianst]

Oh dear. Can you run:

node node_modules\meshcentral --debug webrelay,httpheaders

Or enable webrelay and http headers tracing and send me the result? If it's private info, feel free to mail me that log if you want.

[si458]

do you want it with the original file or the modified file? im using the docker image so will email logs as best i can

[Ylianst]

Anything you can provide is ok. You can modify the file if you like. As long as you can send me the log starting at the point you start doing the web relay until it jams, that would be great.

[si458]

the CSV output generated from the web ui shows all the JSON objects as [object Object], not sure if this is another bug or not?

20:29:15 | httpheaders | GET,   /admin/assets/js/jquery-3.1.1.min.js?load_version=16.0.21.9, [object Object]
20:29:15 | httpheaders | GET, /admin/assets/css/outdatedbrowser.min.css?load_version=16.0.21.9,   [object Object]
20:29:15 | httpheaders | GET, /admin/assets/css/notie.css?load_version=16.0.21.9, [object Object]
20:29:15 | httpheaders | GET,   /admin/assets/less/cache/lessphp_e7c9f0982c3f17a1ea8aaa070dea9afe67e7e536.css,   [object Object]
20:29:15 | httpheaders | GET,   /admin/assets/css/typehead.js-bootstrap3-0.2.3.css?load_version=16.0.21.9,   [object Object]
20:29:15 | httpheaders | GET,   /admin/assets/css/jquery-ui-1.10.3.custom.css?load_version=16.0.21.9, [object   Object]
20:29:15 | httpheaders | GET,   /admin/assets/css/bootstrap-table-reorder-rows.css?load_version=16.0.21.9,   [object Object]
20:29:15 | httpheaders | GET, /admin/assets/css/bootstrap-table-dev.css?load_version=16.0.21.9,   [object Object]
20:29:15 | httpheaders | GET, /admin/assets/css/font-awesome.min-4.7.0.css?load_version=16.0.21.9,   [object Object]
20:29:15 | httpheaders | GET, /admin/assets/css/bootstrap-3.3.7.min.css?load_version=16.0.21.9,   [object Object]
20:29:15 | httpheaders | GET, /admin/config.php, [object Object]
20:29:15 | httpheaders | GET, /admin/, [object Object]
20:29:15 | httpheaders | GET, /admin, [object Object]
20:29:15 | httpheaders | GET, /images/icon-relay-notify10.png, [object Object]

[si458]

not sure if this helps but here is the HTTP header for the text/event-stream if it helps

GET

/admin/ajax.php?module=dashboard&command=netmon

{"host":"relay.meshcentral.myserver.com","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:103.0) Gecko/20100101 Firefox/103.0","accept":"text/event-stream","accept-encoding":"gzip, deflate, br","accept-language":"en-GB","cache-control":"no-cache","cookie":"lang=en_US; searchHide=1; xid=eyJ4IjoieDAzYkVOa1ciLCJ1c2VyaWQiOiJ1c2VyLy9hZG1pbiJ9; PVEAuthCookie=PVE%3Aroot@pam%3A62E5530A%3A%3Ap6jVusk0BcyUWWAeZorGSwAoiTab6jahj2DBcBc1c5XkusKoqFaCaNPTmUv4si/+Q+/mfaBzJHPTLsgwIWTSsvAAeaBUQ6T/82l5kSETeFq3oTev3ltPBZORXq7QsLv0KDUMbqaU065Mv23W4yGEaYYbI1Ac/4vNVW62W2e4gPK/eCNAneu1Vvskihtfh2PDyy//wrAdQzBI66PAJ/g0rg37RHHSq0A+E62jICH1awRm8TBnprmXhpPmsmN7uLTqlkuySvrdqU76JZ6Ls+LYdco9VfTBpy7lrZyY8Z8UzdLMDWKKrU5I6A/TA0TF2Sl4aDVbTDULNByOVAaHVakGLA%3D%3D; xid.sig=zWkMbPdvPONuN3uup4-P9TzH0RvdUWCmiPoLqQ5vY72G-3yJ4O806aEI84FIN6Mq; PHPSESSID=ob84ei3db64pckdf33dfvskkia","pragma":"no-cache","referer":"https://relay.meshcentral.myserver.com/admin/config.php","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","te":"trailers","x-forwarded-for":"217.14.1.1","x-forwarded-host":"relay.meshcentral.myserver.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"94ef1ecbf78a","x-real-ip":"217.14.1.1"}

[Ylianst]

For my use, ran it thru JSON lint...

{
  "host": "relay.meshcentral.myserver.com",
  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:103.0) Gecko/20100101 Firefox/103.0",
  "accept": "text/event-stream",
  "accept-encoding": "gzip, deflate, br",
  "accept-language": "en-GB",
  "cache-control": "no-cache",
  "cookie": "lang=en_US; searchHide=1; xid=eyJ4IjoieDAzYkVOa1ciLCJ1c2VyaWQiOiJ1c2VyLy9hZG1pbiJ9; PVEAuthCookie=PVE%3Aroot@pam%3A62E5530A%3A%3Ap6jVusk0BcyUWWAeZorGSwAoiTab6jahj2DBcBc1c5XkusKoqFaCaNPTmUv4si/+Q+/mfaBzJHPTLsgwIWTSsvAAeaBUQ6T/82l5kSETeFq3oTev3ltPBZORXq7QsLv0KDUMbqaU065Mv23W4yGEaYYbI1Ac/4vNVW62W2e4gPK/eCNAneu1Vvskihtfh2PDyy//wrAdQzBI66PAJ/g0rg37RHHSq0A+E62jICH1awRm8TBnprmXhpPmsmN7uLTqlkuySvrdqU76JZ6Ls+LYdco9VfTBpy7lrZyY8Z8UzdLMDWKKrU5I6A/TA0TF2Sl4aDVbTDULNByOVAaHVakGLA%3D%3D; xid.sig=zWkMbPdvPONuN3uup4-P9TzH0RvdUWCmiPoLqQ5vY72G-3yJ4O806aEI84FIN6Mq; PHPSESSID=ob84ei3db64pckdf33dfvskkia",
  "pragma": "no-cache",
  "referer": "https://relay.meshcentral.myserver.com/admin/config.php",
  "sec-fetch-dest": "empty",
  "sec-fetch-mode": "cors",
  "sec-fetch-site": "same-origin",
  "te": "trailers",
  "x-forwarded-for": "217.14.1.1",
  "x-forwarded-host": "relay.meshcentral.myserver.com",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https",
  "x-forwarded-server": "94ef1ecbf78a",
  "x-real-ip": "217.14.1.1"
}

Oh! This helps a lot. I expected:

Content-Type: text/event-stream

But your request uses "accept".

[Ylianst]

Ok, I got a much improved version, give it another try with this file. Let me know if it works.

[si458]

Ok, I got a much improved version, give it another try with this file. Let me know if it works.

1. the speed of the relay now seems to have improved! (not sure why but go figure)
2. the stupid 'text/event-stream' isnt blocking other requests anymore it would seem
3. HOWEVER im not getting any data from the event-stream on the webpage from the graphs arent being populated?
4. when checking the MC web panel as i flip between the dashboard and say the CDR Log page, the TCP Routing count keeps increasing? BUT never goes down? (so i have to force it closed via the trash can icon)

[Ylianst]

Oh! Just reading your response I know what is going on. I will work on a fix for this.

[Ylianst]

I just built myself a full ExpressJS text/event-stream sample... so, I am finally test this case correctly.

[Ylianst]

This is not good, my test app work correctly thru web-relay. So, I can't replicate the issue.

[Ylianst]

Ok, I did find that I did not handle HTTP socket closing correctly and so, just put in a fix for this. It's in "apprelay.js" so if you want to update just that file and try it, you can do so.

[si458]

Ok, I did find that I did not handle HTTP socket closing correctly and so, just put in a fix for this. It's in "apprelay.js" so if you want to update just that file and try it, you can do so.

sorry for my delay. ok that seems to fix the increasing of the tcp count after every page navigate, HOWEVER im still not getting any data from the event-stream in the web browser

is the a way i can debug the data being sent over that relay? then i can check what data is coming from the agent before being forwarded to the client ?