Ylianst
Memory Leak
Describe the bug Mesh Agent uses more and more RAM over time until the device it's installed on runs out after approximately 12 hours and we have to reboot it. This started some time between October 25th and October 27th and no changes were made by us as far as we know.
To Reproduce Only reproducible on one of our 40 deployed agents.
Expected behavior Expect it to use a steady and low amount of RAM.
Screenshots
In the second screenshot you'll see it using 48.9% of available memory after 5.25 hours. It continues to increase until the 12hr mark as mentioned above.
Server Software (please complete the following information):
- OS: Ubuntu 22.04.5 LTS
- Virtualization: Proxmox
- Network: LAN/WAN
- Version: 1.1.53
- Node: Unsure
Client Device (please complete the following information):
- Device: A different virtual machine than the one above
- OS: Ubuntu 22.04.5 LTS
- Network: Local to Meshcentral
- Browser: Google Chrome
- MeshCentralRouter Version: Unsure
Your config.json file Don't know where to find this, will keep looking and add it later.
Plz can u upload the images to the issue instead of imgur links? (Im UK and imgur has been blocked/banned in the UK now, and i dont pay for a vpn access anywhere)
Strange one but are u having Internet issues at all?
The memory leak we know about (but not sure how to fix yet) happens when the device in question struggles to connect to meshcentral and it retries the connection over and over again
This isn't a meshcentral problem but a meshagent issue (you can check the meshagent repo about this)
Edit: a simple memory fix is to just restart the meshagent every so often service restart (from the console tab of the device in the Web ui)
No internet issues that I'm aware of. Other agents on the same network don't have this problem. Should I post this on the meshagent github?
You can do if you wanted but the is already open issues about this problem in the repo https://github.com/Ylianst/MeshAgent/issues/110 https://github.com/Ylianst/MeshAgent/issues/281 https://github.com/Ylianst/MeshAgent/issues/151
The issue as explained is because the meshagent loses connection to meshcentral and reties over and over again until it reconnects successfully and the is a leak somewhere
You can actually test/see this for yourself, install the agent on a Linux machine, then block its Internet access, then watch the memory and after about 5-10mins it starts climbing...
Sadly we need a c++ developer to look into the code to where its leaking from
The is also a work around too a gent came up with that if its more than 128mb in ram then just restart the service and this uses Linux itself, will reply back when I find link to the fix! https://github.com/Ylianst/MeshAgent/issues/281#issuecomment-3109202154
Also from the screenshot the is a huge gap with no data, also the memory leak doesn't climb so quickly its gradually
So i would say this is the problem, the was an Internet outage with the machine in question and this caused the memory leak in the meshagent sadly
The VM was just turned off during that gap in the screenshot. But I'll look into possible connectivity problems. Thanks.
Sadly we need a c++ developer to look into the code to where its leaking from
Does this mean you don't have a C++ developer?
@DBMTC yes sadly we dont have a C++ developer, MeshCentral is a community driven project/appliation and rely on the community to help it survive and live on! im more of the backend end dev which can fix the meshcentral sever side things we have dan the docker man, and we have a few others who help with UI changes etc (html/css/js) but we sadly dont have anyone who knows c++ for linux/mac/windows etc who can help us fix things thats wrong with the meshagent 👎
I'm more of C than C++, but I'm getting a bit used to it and I would like to have a look into it.
A few questions for @TobinSol-Matt : · Are you using any MeshCentral plugins? [Updated to ping the correct person. Sorry DBMTC!]
I am not using meshcentral at all, I am going to evaluate it.
Can you use valdgrind to start the incriminated service that is memory leaking and reproduce the leaking? Valgrind should be able to give you a clue on what to look for... Ideally this is something that can be solved with a bit of unique_ptr or shared_ptr magic.
I can perhaps help you in getting this fixed...
I am not using meshcentral at all, I am going to evaluate it.
Can you use valdgrind to start the incriminated service that is memory leaking and reproduce the leaking? Valgrind should be able to give you a clue on what to look for... Ideally this is something that can be solved with a bit of unique_ptr or shared_ptr magic.
I can perhaps help you in getting this fixed...
If you are willing to help, you are welcome!
