danog
High server load
Hello. Thanks for the incredibly cool library. Respect. For the company's internal needs, I made an integration with the library, there are only about 12 connections (we tested it with 23), and the server is loaded quite heavily every 5-10 hours. What configuration methods are there to prevent this from happening? I read in the documentation that using a database, the load will not be so great, but still, the swap is almost always full and all the memory is used by all processes, although the accounts are actively used only once every 5 minutes and the scripts themselves are fast.
This is what it looks like in htop:
The settings I use:
public function getApiConnect($account = null): API
{
$path = dirname(__DIR__, 2).'/var/log/';
$settings = (new Settings) // Починаємо з загальних налаштувань
->setAppInfo(
(new AppInfo)
->setApiId($this->parameterBag->get('tdlib_api_id'))
->setApiHash($this->parameterBag->get('tdlib_api_hash'))
)
->setLogger( // Додаємо налаштування логування
(new Settings\Logger())
->setMaxSize(100 * 1024 * 1024) // Максимальний розмір файлу логів 100MB (за бажанням)
->setExtra($path.'MadelineProto.log')
)
->setDb(
(new \danog\MadelineProto\Settings\Database\Mysql)
->setUri('tcp://'.$this->parameterBag->get('danog_host'))
->setDatabase($this->parameterBag->get('danog_database'))
->setUsername($this->parameterBag->get('danog_username'))
->setPassword($this->parameterBag->get('danog_password'))
->setEnableFileReferenceDb(true)
->setEnableMinDb(true)
->setEnableUsernameDb(true)
->setEnableFullPeerDb(true)
->setEnablePeerInfoDb(true)
)
;
return new API($this->getPathSession($account), $settings);
}
By the way, if you don't set SetEnable, for some reason there are problems with access, although they seem to be true by default (observation).
It would be good to have a setting at connection if the process hangs or uses more memory than specified - so that this process is killed. Please show examples of optimization, or errors in my connection. Thanks in advance.
It seems like madelineproto is simply not using the db, are you sure you're using the latest version and are you sure you correctly updated the settings to use the db?
Use "danog/madelineproto": "^8.5.4". Yes, there is a guarantee because the database creates tables and fills them with data. I customized the connection script a little:
public function getApiConnect($account = null): API
{
$path = dirname(__DIR__, 2).'/var/log/';
$settings = (new Settings)
->setConnection(
(new Settings\Connection())
->setPingInterval(120)
->setTimeout(60)
)
->setPeer(
(new Settings\Peer())
->setFullFetch(false)
->setCacheAllPeersOnStartup(false)
->setFullInfoCacheTime(3600) // кешувати деталі довше
)
->setSerialization(
(new Settings\Serialization())
->setInterval(300)
)
->setAppInfo(
(new AppInfo)
->setApiId($this->parameterBag->get('tdlib_api_id'))
->setApiHash($this->parameterBag->get('tdlib_api_hash'))
)
->setLogger(
(new Settings\Logger())
->setMaxSize(100 * 1024 * 1024) // Максимальний розмір файлу логів 100MB (за бажанням)
->setExtra($path.'MadelineProto.log')
)
->setDb(
(new \danog\MadelineProto\Settings\Database\Mysql)
->setUri('tcp://'.$this->parameterBag->get('danog_host'))
->setDatabase($this->parameterBag->get('danog_database'))
->setUsername($this->parameterBag->get('danog_username'))
->setPassword($this->parameterBag->get('danog_password'))
->setEnableFileReferenceDb(true)
->setEnableMinDb(true)
->setEnableUsernameDb(true)
->setEnableFullPeerDb(true)
->setEnablePeerInfoDb(true)
)
;
return new API($this->getPathSession($account), $settings);
}
and added this after it:
$api = $this->getApiConnect($account);
$api->setNoop();
I think it reduced the load, but I'm still testing. It would be cool to have a command to restart all library processes, because they hang in the background and only rebooting the entire server helps. It would also be cool to have a limit on CPU load, that would be super cool.
I would need more logs to properly debug the issue, also try getting a memory profile with https://docs.madelineproto.xyz/PHP/danog/MadelineProto/API.html#getMemoryProfile
Thanks. When the system is under load again, I will make a conclusion about the memory and add it here. If it is not loaded, I will close this issue. Thanks for the feedback.
@danog And here I caught the server load again at 40%+. For some reason, errors started to appear: Sending on the channel failed. Did the context die? And then the load grew to high values. I am attaching the file with getMemoryProfile.
The attached profile is not readable by cachegrind, and there are no logfiles either...
Nvm about the logfile, forgot it requires a legacy version of gperftools, still it actually seems to track allocation sources not current usage so that sucks; either way, the logfile should contain more info.
Hello @danog
- Enabled memprof
- Added MEMPROF_PROFILE=1 to ENV, checked that php definitely sees the MEMPROF_PROFILE parameter.
- Additionally set $settings->getMetrics()->setEnableMemprofCollection(true) Everything that was able to be removed was placed in the archive. result.log.zip
@danog Hello. Sorry for bothering you, did you manage to check my case?
I'll repeat myself of course, but it would be great to have some command or sh script that would restart workers that have reached peak load on the server. Analog restart nginx.
From the logs, it seems simply that the scripts making the request suddenly disconnect, there may be also issues with some requests not getting a reply from telegram in time; could you try splitting the logs of each session, and getting the logs only for the single session that is hanging?
Hello, @danog . The situation is repeating itself again, but I reworked the work with the library separately from Symfony and added some memory and time restrictions, so the script "disappears" under conditions that do not suit me. I could not get the logs, because for some reason, when I turned on memprof, the extension_loaded method does not see it, but it is in phpinfo. While I am trying to get the logs, I have a question: Will the option work when the worker process takes 20%+ of memory, connect to it and restart?
$settings = (new Settings)
->setAppInfo(
(new AppInfo)
->setApiId($this->parameterBag->get('tdlib_api_id'))
->setApiHash($this->parameterBag->get('tdlib_api_hash'))
)
->setLogger(
(new Settings\Logger())
//->setLevel(\danog\MadelineProto\Logger::LEVEL_ULTRA_VERBOSE)
->setMaxSize(100 * 1024 * 1024) // Максимальний розмір файлу логів 100MB (за бажанням)
->setExtra($path."{$sessionCode}.log")
)
->setDb(
(new \danog\MadelineProto\Settings\Database\Mysql)
->setUri('tcp://'.$this->parameterBag->get('danog_host'))
->setDatabase($this->parameterBag->get('danog_database'))
->setUsername($this->parameterBag->get('danog_username'))
->setPassword($this->parameterBag->get('danog_password'))
->setEnableFileReferenceDb(true)
->setEnableMinDb(true)
->setEnableUsernameDb(true)
->setEnableFullPeerDb(true)
->setEnablePeerInfoDb(true)
);
$api = new API($pathSession, $settings);
$api->restart();
I checked, the worker process is closed and a new one is created, but I am not sure that the session will close properly and there will be no problems in the future.
@danog Hello. For one account, this restart is enough for a day, but for another, immediately after the restart, it still takes up a lot of memory, 33% or more. Maybe you should do it every time after finishing work so that the process ends. So far the record is 57% and it's growing so that you constantly need to restart so as not to lose the session or cause the server to reboot (because elastic also takes 20%). The swap is 99% full, almost 1GB.
Is there really no way to limit the running time of processes? For example, if they're not being used? Unfortunately, I couldn't find any information in the documentation about the ability to specify an iddle timeout.
@entermix Go from the other side, set the lifetime of the php script. set_time_limit(300); ini_set('memory_limit', '512M');
But my experience shows that you need to open a session, do some actions and then at the end do: $api->restart() then the process dies and will rise only when you open it next time. And here you do not need additional authorizations, connect to the same session.
@danog Hello. Can you look into my problem? I thought that maybe updating to a new version of the library would save the situation, but there is an error right there about the chat database.
@danog Everything worked for two weeks, but now only this error: The operation was cancelled It is not clear what this is related to, because the script did not change, the number of chats too, and in general everything was done the same.
Versions: "danog/madelineproto": "8.4.18", "danog/async-orm": "1.1.3", "amphp/http-client": "5.3.3"
@entermix Yes, I am also in favor of having the ability to end the process in a better way than restart(), although it does the same thing (but logically you expect the process to restart, but it won't start until there are new requests from PHP).
As mentioned above, on idle, the MadelineProto worker must use ~0% CPU.
What you are seeing is abnormal behavior, a bug, but until you provide a valid reproducer, I cannot help with fixing it
@danog, please tell me why my comment was deleted? Did I do anything wrong?
@danog Hello. Yes, I see that in general the process takes up little memory and the processor is causing the problem, but over time it takes up too much memory and there is no way to restart it at this point, $api->restart() restarts, but still very quickly the memory load increases to 20-30%.
Sorry, at the time of the problem I couldn't make logs, now I tried - the script worked.
These are logs without MEMPROF_PROFILE=1: madelineproto.zip
This is with MEMPROF_PROFILE=1 enabled memory-profile_2.log.zip
I don't use EventLoop, but it seems that at some point it freezes, loops, and hence the increasing memory usage. For me, it is important to connect to the session at the moment, receive messages from the chat and finish the work, and now it turns out that I did everything I needed, and the script continues in the background somehow loads the system, and this is something I cannot control. Any system can fail, perhaps you need a tool that will allow you to set a memory usage limit and "kill" the process yourself, in case something goes wrong.
@danog I increased the server capacity to 4 cores and 12GB of memory, but all the workers (about 23x) occupy them, 30-45% of the memory is the norm for a while, but after that they freeze and do not load. Maybe this is caused by the encryption in front of the database in madelineproto?
понедельник, 20 октября 2025 г. пользователь Oleksandr Kurochka < @.***> написал:
b**yhaskell left a comment (danog/MadelineProto#1661) https://github.com/danog/MadelineProto/issues/1661#issuecomment-3420749603
@danog https://github.com/danog I increased the server capacity to 4 cores and 12GB of memory, but all the workers (about 23x) occupy them, 30-45% of the memory is the norm for a while, but after that they freeze and do not load. Maybe this is caused by the encryption in front of the database in madelineproto?
— Reply to this email directly, view it on GitHub https://github.com/danog/MadelineProto/issues/1661#issuecomment-3420749603, or unsubscribe https://github.com/notifications/unsubscribe-auth/BVAFDJEBVJYRWJD5JIUKZND3YR66XAVCNFSM6AAAAACDPMLJWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTIMRQG42DSNRQGM . You are receiving this because you commented.Message ID: @.***>
-- null
