Ylianst
Servers crashing with multiple users
Describe the bug The title doesn't do it justice...but this is hard to describe. We have four users normally using MeshCentral. Out of the blue this morning one by one each user started having issues logging into the server. After much troubleshooting, the issue appears to be that we can log in but then the server crashes and kills all sessions. Sometimes a single user can stay logged in for hours but then another user logs in and there is a crash. We also have not been able to isolate it to a specific user account, browser, or anything else. We've tried clearing all session data, cookies, history, etc from our browers, including using incognito mode, with no change. We have stayed up to date on versions, and have not had this issue since deploying over a year ago. We did try rolling back to 1.1.44, but the issue still occurs. Testing on 1.1.51 in a test environment and everything seems to be just fine, only affecting production. Authentication is done through an ldap connection, unfortunately can't get in through local auth as email doesn't seem to be working so can't verify the local break-glass account.
To Reproduce Steps to reproduce the behavior: As described above....we have not nailed down specific steps that cause it, but just logging in seems to crash the server.
Expected behavior Normal functioning of the server
Screenshots Here's what's in mesherrors.txt
-------- 10/8/2025, 9:40:48 PM ---- 1.1.51 --------
node:_http_outgoing:699
throw new ERR_HTTP_HEADERS_SENT('set');
^
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
at ServerResponse.setHeader (node:_http_outgoing:699:11)
at ServerResponse.header (/opt/meshcentral/meshcentral/node_modules/express/lib/response.js:794:10)
at ServerResponse.header (/opt/meshcentral/meshcentral/node_modules/express/lib/response.js:797:12)
at handleRootRequestEx (/opt/meshcentral/meshcentral/webserver.js:2957:13)
at Timeout._onTimeout (/opt/meshcentral/meshcentral/webserver.js:1392:44)
at listOnTimeout (node:internal/timers:588:17)
at process.processTimers (node:internal/timers:523:7) {
code: 'ERR_HTTP_HEADERS_SENT'
}
Node.js v22.16.0
Server Software (please complete the following information):
- OS: Rocky Linux 9.6
- Virtualization: Docker
- Network: LAN only, reverse proxy in use in front of Mesh
- Version: 1.1.44, 1.1.50, and 1.1.51 (these are what we've tested)
- Node: What's in docker, in this case on 1.1.51 it's 22.16.0
Client Device (please complete the following information):
- Device: Multiple laptops, Android tablet
- OS: Windows 11, Android
- Network: local, wired and wireless
- Browser: Chrome, Edge, Firefox
- MeshCentralRouter Version: NA
Remote Device (please complete the following information):
- Device: NA, never get that far, this is right after logging in
- OS:
- Network:
- Current Core Version (if known):
Additional context Coincedentally, the issues started around the same time Microsoft had Azure issues this morning. That said, we are not doing any oauth to Azure, just local LDAP connections during auth so I don't think this is related.
Your config.json file
{
"$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
"settings": {
"cert": "mesh.privatedomain.org",
"_WANonly": true,
"_LANonly": true,
"_sessionKey": "thisisprivateinformation",
"port": 443,
"_aliasPort": 443,
"redirPort": 80,
"_redirAliasPort": 80,
"BrowserPong": 2700,
"AgentPong": 2700,
"TLSOffload": "npm",
"_ignoreAgentHashCheck": false,
"SelfUpdate": false,
"AllowFraming": false,
"mariaDB": {
"host": "mariadb",
"user": "meshcentral",
"password": "thisisprivateinformation",
"port": 3306
},
"_autoBackup": false,
"WebRTC": true,
"DesktopMultiplex": true,
"maxInvalidLogin": {
"time": 5,
"count": 5,
"coolofftime": 30
},
"maxInvalid2fa": {
"time": 5,
"count": 5,
"coolofftime": 30
}
},
"domains": {
"": {
"title": "MeshCentral",
"_title2": "meshcentral",
"titlePicture": "TitlePicture.png",
"welcomeText": "MeshCentral is a service operated by the Information Technology team, please direct any questions to [email protected].",
"welcomePicture": "Building.png",
"welcomePictureFullScreen": true,
"_footer": "This is a footer test",
"_loginfooter": "This is a login footer test",
"_minify": true,
"NewAccounts": false,
"_userNameIsEmail": true,
"certUrl": "https://mesh.privatedomain.org:443",
"agentInviteCodes": true,
"_maxDeviceView": 50,
"userSessionIdletimeout": 240,
"twoFactorCookieDurationDays": 7,
"passwordRequirements": {
"min": 8,
"max": 128,
"upper": 1,
"lower": 1,
"numeric": 1,
"nonalpha": 1,
"reset": 90,
"force2factor": true,
"_skip2factor": "127.0.0.1,192.168.2.0/24",
"_oldPasswordBan": 5,
"_banCommonPasswords": false,
"twoFactorTimeout": 300
},
"auth": "ldap",
"ldapOptions": {
"url": "ldaps://privatedomain.org:636/",
"bindDN": "CN=svc_meshcentral,CN=Service Accounts,DC=privatedomain,DC=org",
"bindCredentials": "thisisprivateinformation",
"searchBase": "DC=privatedomain,DC=org",
"searchFilter": "(&(objectClass=user)(objectCategory=person)(userPrincipalName={{username}})(!(userAccountControl:1.2.840.113556.1.4.803:=2)))",
"reconnect": true
},
"ldapUserKey": "sAMAccountName",
"ldapUserBinaryKey": "objectGUID",
"ldapUserName": "name",
"ldapUserEmail": "mail",
"ldapUserRealname": "{{{givenName}}} {{{sn}}}",
"ldapUserPhoneNumber": "telephoneNumber",
"ldapUserImage": "thumbnailPhoto",
"ldapUserGroups": "memberOf",
"ldapSyncWithUserGroups": { "filter": [ "OU=MeshCentral" ] },
"ldapUserRequiredGroupMembership": [ "CN=G-MeshCentralAdmins,OU=MeshCentral,OU=Groups,DC=privatedomain,DC=org" ],
"ldapSiteAdminGroups": [ "CN=G-MeshCentralAdmins,OU=MeshCentral,OU=Groups,DC=privatedomain,DC=org" ]
}
},
"smtp": {
"host": "smtp.privatedomain.org",
"port": 25,
"from": "[email protected]",
"__tls__": "When 'tls' is set to true, TLS is used immidiatly when connecting. For SMTP servers that use TLSSTART, set this to 'false' and TLS will still be used.",
"tls": false,
"___tlscertcheck__": "When set to false, the TLS certificate of the SMTP server is not checked.",
"_tlscertcheck": false,
"__tlsstrict__": "When set to true, TLS cypher setup is more limited, SSLv2 and SSLv3 are not allowed.",
"_tlsstrict": true,
"_emailDelaySeconds": 300,
"verifyemail": false
},
"_letsencrypt": {
"__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before>",
"_email": "[email protected]",
"_names": "myserver.mydomain.com",
"production": false
}
}
Additional note....I rolled back to 1.1.35 as that release was still on Node 20.x.x for docker.....but no change in the issue.
-------- 10/8/2025, 10:21:29 PM ---- 1.1.35 --------
node:_http_outgoing:659 throw new ERR_HTTP_HEADERS_SENT('set'); ^
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client at ServerResponse.setHeader (node:_http_outgoing:659:11) at ServerResponse.header (/opt/meshcentral/meshcentral/node_modules/express/lib/response.js:794:10) at ServerResponse.header (/opt/meshcentral/meshcentral/node_modules/express/lib/response.js:797:12) at handleRootRequestEx (/opt/meshcentral/meshcentral/webserver.js:2896:13) at Timeout._onTimeout (/opt/meshcentral/meshcentral/webserver.js:1364:44) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) { code: 'ERR_HTTP_HEADERS_SENT' }
Node.js v20.15.1
UPDATE
In looking at outside environmental changes to our environment, the only thing we have changed is a scheme update to our Active Directory. This update occured in very close timing to when we started having issues. With nothing else to go on, I switched to local authentation from our norm of using LDAP and created local accounts for our team. Thus far we have everyone logged into Mesh and are not experiencing any issues. We do know the test environment has not had it's AD schema updated so time permitting we'll be testing that theory in the test environment, but for now are going to run with local accounts for a few days to feel confident that we're able to work without issue using local accounts.
Hi,
if I understand the webserver.js correctly, line 2957 is:
handleRootRequestEx(req, res, domain, direct);
and before it is checking the user object:
var user = obj.users[userid];
if ((err == null) && checkUserOneTimePasswordRequired(domain, user, req, loginOptions) == true) {
....
} else if ((userid != null) && (err == null)) {
and then calling handleRootRequestEx
but I am not sure. can you maybe enable web debugging, or do some console.log on the way to that call? Sorry I am not a better help
if you can provide me access to the meshcentral serve in question having the issue i can have a look whats going wrong just give me a fleemail/disc/whats/tele/sign 👍
