Ylianst
MeshAgent creating powershell.exe handles
On a Win10 virtual machine (on ESX) I have an issue with the Mesh Agent creating many powershell.exe instances. This will slowly fill up the VM's memory until no virtual memory is left. The memory clears when the meshcentral is restarted.
Any ideas what could be causing this? I do not see any remote sessions in meshcentrals log for this host since the 24th of august and the memory consumption has been gradually growing since that date because of the many powershell.exe instances (see screenshot below).
Meshcentral version: 1.1.27
Host info: OS Name: Microsoft Windows 10 Pro OS Version: 10.0.18362 N/A Build 18362
MeshAgent info:
ooo this is interesting! can you check what command the powershell is running at all? as yes it should run powershell to do a some command etc but should then close afterwards! so might be a bug not closing the powershell session afterwards
ah frig! thats what i was worried about! we have no idea what command its running because we pipe it into the powershell! BUT im sure i remember seeing this issue in the past and it was something to do with Languages in the powershell? what language is your OS?
Uninstalled the agent from this vm for now (wasnt really required on it and it seems like a somewhat isolated issue). Let me know if I can do anything to help debug the issue. I'll be glad to run some tests to fix this.
i found the commits i was talking about and it was to do with the bitlocker https://github.com/Ylianst/MeshCentral/pull/5832 https://github.com/Ylianst/MeshCentral/pull/5747 can you verify if the VM in question was using bitlocker and was it enabled at all?
edit: can you also share a screenshot of the details tab? does it show information missing at all?
Bitlocker is actually not enabled on this VM:
PS C:\WINDOWS\system32> manage-bde -status
BitLocker Drive Encryption: Configuration Tool version 10.0.18362
Copyright (C) 2013 Microsoft Corporation. All rights reserved.
Disk volumes that can be protected with
BitLocker Drive Encryption:
Volume C: []
[OS Volume]
Size: 59,05 GB
BitLocker Version: None
Conversion Status: Fully Decrypted
Percentage Encrypted: 0,0%
Encryption Method: None
Protection Status: Protection Off
Lock Status: Unlocked
Identification Field: None
Key Protectors: None Found
I have reinstalled the agent and these show up as details:
All looks OK, very weird? Does it now show multiple powershells at all after u reinstalled it?
Does it happen straight away right after u install the meshagent Or does it appear after a period of time?
Also one thing to try is watch it in process explorer (Count number of powershells)
Then run sysinfo from the console tab of the device,
Wait for it to return data, and see if more powershells appear, if they do then we have tracked down what it is!
Straight away immediately after launch I see multiple powershell.exe processes spawning, only one of which persists and presumably fails to exit. Thats consistent behavior both before and after I reinstalled it. Then, with patience (I did not time it at all but I think like every 10-30mins or so) another powershell.exe process gets added to the process list.
Yep, that's the sysinfo !
Every 30ish mins, it will do a refresh of ur specs of ur computer!
One way to trigger it manually is to run sysinfo in the console tab of the device in the Web ui
If u have a moment and happy too, email me the output from sysinfo and I can compare with a working system and see what's missing.
Then the missing information will be the culprit of the rogue powershell
Here ya go, hope this helps and is what you meant (I redacted some serialno's/uid's with 000-000-0000 like strings).
> sysinfo
{
"hardware": {
"windows": {
"memory": [
{
"BankLabel": "RAM slot #0",
"Capacity": "17179869184",
"Caption": "Fysiek geheugen",
"CreationClassName": "Win32_PhysicalMemory",
"DataWidth": 32,
"Description": "Fysiek geheugen",
"DeviceLocator": "RAM slot #0",
"FormFactor": 8,
"MemoryType": 2,
"Name": "Fysiek geheugen",
"SMBIOSMemoryType": 3,
"Tag": "Physical Memory 0",
"TotalWidth": 32,
"TypeDetail": 512
}
],
"osinfo": {
"BootDevice": "\\Device\\HarddiskVolume1",
"BuildNumber": "18362",
"BuildType": "Multiprocessor Free",
"Caption": "Microsoft Windows 10 Pro",
"CodeSet": "1252",
"CountryCode": "31",
"CreationClassName": "Win32_OperatingSystem",
"CSCreationClassName": "Win32_ComputerSystem",
"CSName": "BATCH",
"CurrentTimeZone": 120,
"DataExecutionPrevention_32BitApplications": true,
"DataExecutionPrevention_Available": true,
"DataExecutionPrevention_Drivers": true,
"DataExecutionPrevention_SupportPolicy": 2,
"Description": "batch",
"EncryptionLevel": 256,
"ForegroundApplicationBoost": 2,
"InstallDate": "20240706174958.000000+120",
"LastBootUpTime": "20240829155423.500000+120",
"Locale": "0413",
"Manufacturer": "Microsoft Corporation",
"MaxNumberOfProcesses": -1,
"Name": "Microsoft Windows 10 Pro|C:\\WINDOWS|\\Device\\Harddisk0\\Partition2",
"NumberOfProcesses": 180,
"NumberOfUsers": 9,
"OperatingSystemSKU": 48,
"OSArchitecture": "64 bits",
"OSLanguage": 1043,
"OSProductSuite": 256,
"OSType": 18,
"Primary": true,
"ProductType": 1,
"RegisteredUser": "Windows-gebruiker",
"SerialNumber": "00000-00000-00000-00000",
"SizeStoredInPagingFiles": "7740164",
"Status": "OK",
"SuiteMask": 272,
"SystemDevice": "\\Device\\HarddiskVolume2",
"SystemDirectory": "C:\\WINDOWS\\system32",
"SystemDrive": "C:",
"Version": "10.0.18362"
},
"partitions": [
{
"BlockSize": "512",
"Bootable": true,
"BootPartition": true,
"Caption": "Schijfnr. 0, partitienr. 0",
"CreationClassName": "Win32_DiskPartition",
"Description": "Installable File System",
"DeviceID": "Disk #0, Partition #0",
"Name": "Schijfnr. 0, partitienr. 0",
"NumberOfBlocks": "1024000",
"PrimaryPartition": true,
"Size": "524288000",
"StartingOffset": "1048576",
"SystemCreationClassName": "Win32_ComputerSystem",
"SystemName": "BATCH"
},
{
"BlockSize": "512",
"Caption": "Schijfnr. 0, partitienr. 1",
"CreationClassName": "Win32_DiskPartition",
"Description": "Installable File System",
"DeviceID": "Disk #0, Partition #1",
"Index": 1,
"Name": "Schijfnr. 0, partitienr. 1",
"NumberOfBlocks": "123842678",
"PrimaryPartition": true,
"Size": "63407451136",
"StartingOffset": "525336576",
"SystemCreationClassName": "Win32_ComputerSystem",
"SystemName": "BATCH"
},
{
"BlockSize": "512",
"Caption": "Schijfnr. 0, partitienr. 2",
"CreationClassName": "Win32_DiskPartition",
"Description": "Unknown",
"DeviceID": "Disk #0, Partition #2",
"Index": 2,
"Name": "Schijfnr. 0, partitienr. 2",
"NumberOfBlocks": "952320",
"PrimaryPartition": true,
"Size": "487587840",
"StartingOffset": "63933775872",
"SystemCreationClassName": "Win32_ComputerSystem",
"SystemName": "BATCH"
}
],
"cpu": [
{
"Caption": "Intel64 Family 6 Model 85 Stepping 7",
"DeviceID": "CPU0",
"Manufacturer": "GenuineIntel",
"MaxClockSpeed": 2993,
"Name": "Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz",
"SocketDesignation": "CPU #000"
},
{
"Caption": "Intel64 Family 6 Model 85 Stepping 7",
"DeviceID": "CPU1",
"Manufacturer": "GenuineIntel",
"MaxClockSpeed": 2993,
"Name": "Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz",
"SocketDesignation": "CPU #001"
}
],
"gpu": [
{
"Name": "VMware SVGA 3D",
"CurrentHorizontalResolution": 2184,
"CurrentVerticalResolution": 864
}
],
"drives": [
{
"Caption": "VMware Virtual disk SCSI Disk Device",
"DeviceID": "\\\\.\\PHYSICALDRIVE0",
"Model": "VMware Virtual disk SCSI Disk Device",
"Partitions": 3,
"Size": "64420392960",
"Status": "OK"
}
],
"volumes": {
"A": {
"type": "Unknown",
"size": 0,
"sizeremaining": 0,
"removable": true
},
"C": {
"type": "NTFS",
"size": 63407448064,
"sizeremaining": 23507181568
},
"D": {
"type": "Unknown",
"size": 0,
"sizeremaining": 0,
"cdrom": true
}
}
},
"identifiers": {
"bios_date": "20201112000000.000000+000",
"bios_vendor": "Phoenix Technologies LTD",
"bios_version": "6.00",
"bios_serial": "VMware-42 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00",
"bios_mode": "Legacy",
"board_name": "440BX Desktop Reference Platform",
"board_vendor": "Intel Corporation",
"product_uuid": "00000000-0000-0000-0000-000000000000",
"product_name": "VMware Virtual Platform",
"gpu_name": [
"VMware SVGA 3D"
],
"storage_devices": [
{
"Caption": "VMware Virtual disk SCSI Disk Device",
"Model": "VMware Virtual disk SCSI Disk Device",
"Size": "64420392960"
}
],
"cpu_name": "Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz"
},
"agentvers": {
"openssl": "1.1.1s",
"duktape": "v2.6.0",
"commitDate": "2022-12-02T19:42:16.000Z",
"commitHash": "1ac56a2aa26cc403efe758fc53386a3dc0e706b7",
"compileTime": "12:12:34, Dec 9 2022"
},
"network": {
"dns": [
"10.1.1.1",
"10.1.1.2"
]
}
},
"pendingReboot": null,
"hash": "A0810A47D335B7A2A57137B47EC96679819E711EBE3F6D11481E02604F3A45EB50A5F3A545230DABE502F72C2E94C2BF"
}
Some more info;
- After a manual
sysinfocommand another stale powershell.exe process does -not- get stuck and added to the process list. I verified this by running the powershell cmd@(get-process -ea silentlycontinue "powershell").countbefore and after sysinfo. During sysinfo it will go +1 and when the output is shown in the console tab it will go -1. - If I have just restarted the meshagent the sysinfo will output its result within under 5 seconds. When I tried running
sysinfoon the agent process which I reinstalled and started just this morning, it would take much much longer to output results.
Not sure if this helps but I can imagine it is relevant info
Also,
- Starting either a new powershell-administrator session on the console tab or desktop session in meshcentral on this host took a long while (powershell terminal would just say "connected" and not show terminal output. desktop session would show "connected" and show a black screen for a while)
- After restarting the agent on the host, everything was snappy and responsive again on starting new sessions.
By the way as this is the only host on my ~200 agent installs, (a few of which are on windows server vm's instead of win10 vm in the same ESX infrastructure) which seemingly is showing this behavior I also ran a sfc /scannow and DISM /Online /Cleanup-Image /ScanHealth command to rule out problems originating from invalid windows binaries.
I'm not able to replicate this issue, but I see that the windows_volumes uses a different line terminator than all other uses of powershell: https://github.com/Ylianst/MeshAgent/blob/90f730c3d49555e5c16106a7c8dd0219818edad6/modules/identifiers.js#L372
It uses \n instead of \r\n. It looks like your volumes are returning, and it should only return if that powershell exits, but it does run on startup and is part of the sysinfo, so that might be a place to look for the issue, Simon.
Is the information returned by the volumes accurate for that device?
@HuFlungDu the volume stuff is now returned by the computer-identifiers.js on the meshcentral server itself
So would need to update there if the is a bug with volumes
@HuFlungDu ive just pushed the fix in the meshcentral repo as i do believe you are right! https://github.com/Ylianst/MeshCentral/commit/cf23a3df81de1e1b049a585c72b1b43ecaf40b69
its starting a new line but never actually pressing the enter key as it where to exit the powershell!
we do need to also replace all the inline powershell commands at some point (on my to do list)
replacing them with rather then writing, they run the command directly using -Command
then when something like this happens,
we can see what command was actually running and blocking things
@JasperE84 if you are happy too,
replace the file agents/modules_meshcore/computer-identifiers.js with the master from the meshcentral repo
https://github.com/Ylianst/MeshCentral/blob/master/agents/modules_meshcore/computer-identifiers.js
then restart meshcentral
and see if the issue still happens!
@JasperE84 1.1.29 was just released which included the fix above, can you try upgrading your server and then seeing if the issue is resolved?
Thanks, I've just upgraded the docker container to see if it is fixed and will let you know soon. ~~By the way the hanging powershell.exe instances seem to terminate when the meshserver (not meshagent) is restarted after a docker update.~~
@JasperE84 that's an interesting find... Let us know if it's fixed with the latest 1.1.29 update
Unfortunately 3 stale powershell.exe processes within 40mins. Disregard my statement that the powershell.exe's clear when meshserver is restarted that is not the case. In fact, each meshcentral restart another powershell.exe instance is created. (reproducable on this box).
can you share your config.json at all?
what happens if you run the sysinfo again from the console tab of the device in the web ui?
does a new powershell.exe appear then stay there?
can u also share you info from the server console tab in the web ui too?
has info like what version u running and nodejs etc.
Interesting find, restarting meshserver results in extra powershell.exe hanging.
After meshagent restart:
After subsequent meshserver restart:
But running sysinfo does not
After meshagent restart
After subsequent
sysinfo on agent console tab
> info
Current Core: Dec 9 2022, 100151277
Agent Time: 2024-09-09 16:07:10.812+02:00.
User Rights: 0xffffffff.
Platform: win32.
Capabilities: 15.
Server URL: wss://#########.#############:####/agent.ashx.
OS: Microsoft Windows 10 Pro - 18362.
Modules: amt-apfclient, amt-lme, amt-manage, amt-mei, computer-identifiers, monitor-border, smbios, sysinfo, util-agentlog, wifi-scanner-windows, wifi-scanner, win-console, win-deskutils, win-info, win-securitycenter, win-terminal, win-virtual-terminal, win-volumes.
Server Connection: true, State: 1.
Node ID: ############
Application Location: C:\Program Files\Mesh Agent\
> info
{
"meshVersion": "v1.1.29",
"nodeVersion": "v20.15.1",
"runMode": "Hybrid (LAN + WAN) mode",
"productionMode": true,
"database": "MongoDB",
"dbChangeStream": false,
"dbBulkOperations": false,
"platform": "linux",
"arch": "x64",
"pid": 17,
"uptime": 446.138698989,
"cpuUsage": {
"user": 5246297,
"system": 1031461
},
"memoryUsage": {
"rss": 118198272,
"heapTotal": 49451008,
"heapUsed": 47296632,
"external": 51325021,
"arrayBuffers": 49075685
},
"warnings": [],
"allDevGroupManagers": []
}
$ cat config.json
{
"$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
"settings": {
"plugins":{"enabled": false},
"mongoDb": "mongodb://mongodbadmin:#####@mongodb:27017",
"cert": "mcentral.#####.##",
"_WANonly": true,
"_LANonly": true,
"sessionKey": "#############",
"port": 4430,
"aliasPort": 443,
"agentAliasPort": 4021,
"agentAliasDNS": "mcentral-agent.#######.##",
"redirPort": 800,
"_redirAliasPort": 80,
"AgentPong": 300,
"TLSOffload": true,
"SelfUpdate": false,
"AllowFraming": false,
"WebRTC": false
},
"domains": {
"": {
"title": "######",
"_title2": "docker",
"welcomeText": "Login credentials only available for Sysadmins.<br />Are you a supplier or vendor? Please contact your contact to retrieve a temporary link for support tasks.",
"minify": true,
"NewAccounts": false,
"localSessionRecording": false,
"_userNameIsEmail": true,
"certUrl": "https://mcentral.########.##:443",
"SessionRecording": {
"onlySelectedDeviceGroups": false,
"filepath": "/opt/meshcentral/meshcentral-data/session_recordings",
"index": true,
"maxRecordingSizeMegabytes": 2048,
"protocols": [1,2]
}
}
},
"_letsencrypt": {
"__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before>",
"_email": "[email protected]",
"_names": "myserver.mydomain.com",
"production": false
}
}
thank you @JasperE84 i just wanted to check
- wasnt anything stupid in your config.json (it wasnt 👍)
- wasnt anything stupid like old nodejs (it wasnt 👍)
- and you was 110% on the latest version (which you are 👍)
please can you now try these few things for me now as ur on the latest release
- connect to admin powershell in meshcentral for me (right click connect and click admin powershell)
- run
whoamicheck its output (should say nt authority\system) - run
Get-Volume | Select-Object -Property DriveLetter,FileSystemLabel,FileSystemType,Size,SizeRemaining,DriveType | ConvertTo-Csv -NoTypeInformationand let me have its output - run
Get-BitLockerVolume | Select-Object -Property MountPoint,VolumeStatus,ProtectionStatus | ConvertTo-Csv -NoTypeInformationand let us have its output - can you go into the console tab of the device in meshcentral and run
avand let us have its reply? (it might take a min to return but no more)
Sure, here you go!
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform PowerShell https://aka.ms/pscore6
PS C:\Program Files\Mesh Agent> whoami
nt authority\system
PS C:\Program Files\Mesh Agent> Get-Volume | Select-Object -Property DriveLetter,FileSystemLabel,FileSystemType,Size,SizeRemaining,DriveType | ConvertTo-Csv -NoTypeInformation
"DriveLetter","FileSystemLabel","FileSystemType","Size","SizeRemaining","DriveType"
,"Door systeem gereserveerd","NTFS","524283904","485818368","Fixed"
"C","","NTFS","63407448064","23255228416","Fixed"
,"","NTFS","487583744","70332416","Fixed"
"D","","Unknown","0","0","CD-ROM"
"A","","Unknown","0","0","Removable"
PS C:\Program Files\Mesh Agent> Get-BitLockerVolume | Select-Object -Property MountPoint,VolumeStatus,ProtectionStatus | ConvertTo-Csv -NoTypeInformation
"MountPoint","VolumeStatus","ProtectionStatus"
"C:","FullyDecrypted","Off"
PS C:\Program Files\Mesh Agent>
> av
[
{
"product": "CrowdStrike Falcon Sensor",
"updated": true,
"enabled": true
}
]
All looks OK, but just out of curiosity
What does this output?
U will need to use cmd not powershell in meshcentral web ui
manage-bde -protectors -get C: -Type recoverypassword