Worker container processes reach PHP memory limit and crash

[graham-web]

Trying to debug a small percentage of messages not being sent...

I'm noticing a lot of the following error in the logs for my "worker" container:

PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 335872 bytes) in /var/www/html/vendor/doctrine/dbal/src/Driver/PDO/Statement.php on line 130

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 335872 bytes) in /var/www/html/vendor/doctrine/dbal/src/Driver/PDO/Statement.php on line 130

In Statement.php line 130:

Error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 335872 bytes)

messenger:consume [-l|--limit LIMIT] [-f|--failure-limit FAILURE-LIMIT] [-m|--memory-limit MEMORY-LIMIT] [-t|--time-limit TIME-LIMIT] [--sleep SLEEP] [-b|--bus BUS] [--queues QUEUES] [--no-reset] [--] [...] 2025-05-01 13:43:01,720 WARN exited: messenger-consume-email_00 (exit status 255; not expected) 2025-05-01 13:43:02,747 INFO spawned: 'messenger-consume-email_00' with pid 332

Should the default supervisor command line use the --memory-limit option to prevent this?

[henmohr]

Hey thanks for your reporting.

A hyphothesis is that messenger:consume is spawning multiple times OR your system is out of memory. Can you pass the parameter --time-limit=50 and see if the error persists?

Yes, i guess limiting memory usage it's a good practice too.

Could you test and report back here?

Thanks!

[graham-web]

The system isn't running out of memory, and the container keeps running. The log snippet I shared shows the messenger process for the email queue hitting the PHP memory_limit, then supervisor respawning it.

$ php -i | grep memory_limit memory_limit => 512M => 512M

While sending to a newsletter of ~3000 today, this happened 6 times, approximately 90 seconds between each crash.

Just to note, here is Symfony's documentation on configuring this for production https://symfony.com/doc/5.x/messenger.html#deploying-to-production

I've put --memory_limit=256M on this instance for the moment. Waiting to see what happens when the next message goes out next week.

[O-Mutt]

Hey thanks for your reporting.

A hyphothesis is that messenger:consume is spawning multiple times OR your system is out of memory. Can you pass the parameter --time-limit=50 and see if the error persists?

Yes, i guess limiting memory usage it's a good practice too.

Could you test and report back here?

Thanks!

That should only occur if the messenger:consume is in the cron and not the worker. The more recent image should prevent the first case

[cibero42]

Hello @graham-web ,

Could you please do a docker compose pull to ensure that you are running the latest build?

Furthermore, could you please mention which tag you are using?

[christoph93]

I'm facing a similar issue on 5.2.5-apache.

mautic-cron    | Rebuilding contacts for segment 7
mautic-cron    | 0 total contact(s) to be added in batches of 300
mautic-cron    | 0 total contact(s) to be removed in batches of 300
mautic-cron    | 0 contact(s) affected
mautic-cron    | Rebuilding contacts for segment 17
mautic-cron    | 0 total contact(s) to be added in batches of 300
mautic-cron    | 0 total contact(s) to be removed in batches of 300
mautic-cron    | 0 contact(s) affected
mautic-web     | 127.0.0.1 - - [11/May/2025:02:23:05 +0000] "GET / HTTP/1.1" 302 665 "-" "curl/7.88.1"
mautic-cron    | [11-May-2025 02:23:05 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/docroot/app/bundles/LeadBundle/Entity/CustomFieldRepositoryTrait.php on line 100
mautic-cron    | 
mautic-cron    | Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/docroot/app/bundles/LeadBundle/Entity/CustomFieldRepositoryTrait.php on line 100
mautic-cron    | [11-May-2025 02:23:05 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/vendor/monolog/monolog/src/Monolog/DateTimeImmutable.php on line 1
mautic-cron    | 
mautic-cron    | Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/vendor/monolog/monolog/src/Monolog/DateTimeImmutable.php on line 1

I've set the memory_limit to 4094M on a VPS with 8gb of RAM . Is there something else I can try?

[O-Mutt]

I'm facing a similar issue on 5.2.5-apache.

mautic-cron    | Rebuilding contacts for segment 7
mautic-cron    | 0 total contact(s) to be added in batches of 300
mautic-cron    | 0 total contact(s) to be removed in batches of 300
mautic-cron    | 0 contact(s) affected
mautic-cron    | Rebuilding contacts for segment 17
mautic-cron    | 0 total contact(s) to be added in batches of 300
mautic-cron    | 0 total contact(s) to be removed in batches of 300
mautic-cron    | 0 contact(s) affected
mautic-web     | 127.0.0.1 - - [11/May/2025:02:23:05 +0000] "GET / HTTP/1.1" 302 665 "-" "curl/7.88.1"
mautic-cron    | [11-May-2025 02:23:05 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/docroot/app/bundles/LeadBundle/Entity/CustomFieldRepositoryTrait.php on line 100
mautic-cron    | 
mautic-cron    | Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/docroot/app/bundles/LeadBundle/Entity/CustomFieldRepositoryTrait.php on line 100
mautic-cron    | [11-May-2025 02:23:05 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/vendor/monolog/monolog/src/Monolog/DateTimeImmutable.php on line 1
mautic-cron    | 
mautic-cron    | Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/vendor/monolog/monolog/src/Monolog/DateTimeImmutable.php on line 1

I've set the memory_limit to 4094M on a VPS with 8gb of RAM . Is there something else I can try?

Could you give us some more info on what your setup is? Understanding what crons are running would be helpful here too. A command like docker compose exec -u www-data -w /var/www/html mautic_cron crontab -l should print out what is supposed to be running

[christoph93]

@O-Mutt these are cron jobs. I believe they're the default ones that come with the docker image.

* * * * * php /var/www/html/bin/console mautic:segments:update 2>&1 | tee /tmp/stdout
* * * * * php /var/www/html/bin/console mautic:campaigns:update 2>&1 | tee /tmp/stdout
* * * * * php /var/www/html/bin/console mautic:campaigns:trigger 2>&1 | tee /tmp/stdout
* * * * * php /var/www/html/bin/console mautic:contacts:scheduled_export 2>&1 | tee /tmp/stdout

[cibero42]

@O-Mutt these are cron jobs. I believe they're the default ones that come with the docker image.

* * * * * php /var/www/html/bin/console mautic:segments:update 2>&1 | tee /tmp/stdout
* * * * * php /var/www/html/bin/console mautic:campaigns:update 2>&1 | tee /tmp/stdout
* * * * * php /var/www/html/bin/console mautic:campaigns:trigger 2>&1 | tee /tmp/stdout
* * * * * php /var/www/html/bin/console mautic:contacts:scheduled_export 2>&1 | tee /tmp/stdout

Hi @christoph93 ! You don't seem to be running the latest build from Mautic

Could you please ensure that you're running the tag 5-apache across containers and execute docker container pull?

Thanks!

[christoph93]

Thanks for your reply @cibero42 , I'm now using 5-apache but it didn't fix the issue:

mautic-cron    | 0 total contact(s) to be removed in batches of 300
mautic-cron    | 
mautic-cron    | 
mautic-web     | 172.19.0.3 - - [26/May/2025:02:13:02 +0000] "GET /media/images/logo.png HTTP/1.1" 200 1128812 "-" "-"
mautic-cron    |   9/21 [============>---------------]  42%[26-May-2025 02:13:02 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 44625859 bytes) in /var/www/html/vendor/doctrine/dbal/src/Driver/PDO/Statement.php on line 130
mautic-cron    | 
mautic-cron    | Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 44625859 bytes) in /var/www/html/vendor/doctrine/dbal/src/Driver/PDO/Statement.php on line 130
mautic-cron    | 
mautic-cron    | In Statement.php line 130:
mautic-cron    |                                                                                
mautic-cron    |   Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate   
mautic-cron    |   44625859 bytes)                                                              
mautic-cron    |                                                                                
mautic-cron    | 
mautic-cron    | mautic:campaigns:trigger [-i|--campaign-id [CAMPAIGN-ID]] [--campaign-limit [CAMPAIGN-LIMIT]] [--contact-id [CONTACT-ID]] [--contact-ids [CONTACT-IDS]] [--min-contact-id [MIN-CONTACT-ID]] [--max-contact-id [MAX-CONTACT-ID]] [--thread-id [THREAD-ID]] [--max-threads [MAX-THREADS]] [--kickoff-only] [--scheduled-only] [--inactive-only] [-l|--batch-limit [BATCH-LIMIT]] [-d|--exclude [EXCLUDE]] [--bypass-locking] [-t|--timeout TIMEOUT] [-x|--lock_mode LOCK_MODE] [-f|--force]
mautic-cron    | 
mautic-web     | [Mon May 26 02:13:04.797838 2025] [php:notice] [pid 114:tid 114] [client 127.0.0.1:36936] PHP Deprecation - Creation of dynamic property Mautic\\Middleware\\MiddlewareBuilder::$specs is deprecated - in file /var/www/html/docroot/app/middlewares/MiddlewareBuilder.php - at line 24
mautic-web     | 127.0.0.1 - - [26/May/2025:02:13:04 +0000] "GET / HTTP/1.1" 302 665 "-" "curl/7.88.1"
mautic-cron    | [26-May-2025 02:13:05 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/docroot/app/bundles/LeadBundle/Entity/CustomFieldRepositoryTrait.php on line 100
mautic-cron    | 
mautic-cron    | Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/docroot/app/bundles/LeadBundle/Entity/CustomFieldRepositoryTrait.php on line 100
mautic-cron    | 
mautic-cron    | [26-May-2025 02:13:05 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/vendor/symfony/string/UnicodeString.php on line 1
mautic-cron    | 
mautic-cron    | Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/vendor/symfony/string/UnicodeString.php on line 1
mautic-web     | [Mon May 26 02:13:09.933269 2025] [php:notice] [pid 29:tid 29] [client 127.0.0.1:36946] PHP Deprecation - Creation of dynamic property Mautic\\Middleware\\MiddlewareBuilder::$specs is deprecated - in file /var/www/html/docroot/app/middlewares/MiddlewareBuilder.php - at line 24
mautic-web     | 127.0.0.1 - - [26/May/2025:02:13:09 +0000] "GET / HTTP/1.1" 302 665 "-" "curl/7.88.1"
mautic-web     | [Mon May 26 02:13:15.061555 2025] [php:notice] [pid 30:tid 30] [client 127.0.0.1:35388] PHP Deprecation - Creation of dynamic property Mautic\\Middleware\\MiddlewareBuilder::$specs is deprecated - in file /var/www/html/docroot/app/middlewares/MiddlewareBuilder.php - at line 24
mautic-web     | 127.0.0.1 - - [26/May/2025:02:13:15 +0000] "GET / HTTP/1.1" 302 665 "-" "curl/7.88.1"

Now the logs are also filled with the deprecated message.

Still showing 4gb allowed memory

root@b3e1cb32555d:/var/www/html/docroot# php -ini | grep memory
memory_limit => 4096M => 4096M

Could this have something to do with image sizes? We're using fairly large images in e-mails and the memory limit error show right after it gets the logo.png.

[cibero42]

Could this have something to do with image sizes? We're using fairly large images in e-mails and the memory limit error show right after it gets the logo.png.

I don't really think so, unless your image is very very big.

But have you tried doing the same with a smaller image (< 500 KiB)? Does it work?

What's the size of your logo.png?

[christoph93]

@cibero42 the logo is only 28K.

I thought the memory issue wasn't critical, as Mautic was still sending mail normally. However, today we needed to send a segment e-mail to 130 contacts, and it completely broke Mautic. The logs fill with memory exhausted errors and the send screen stays stuck at 0. I have to restart the stack to get things working again.

After I restart the stack it seems that something does happen while it's stuck, because some mails are queued and most of them just disappear:

At this point, I'm considering migrating to a standard Mautic installation.

[cibero42]

Hello @christoph93,

Could you please try erasing

MAUTIC_MESSENGER_DSN_EMAIL
MAUTIC_MESSENGER_DSN_HIT

From your enviroments?

There might be a bug related to setting those envs. Not sure if it would solve your issue, but it helps nailing down what's happening to your setup.

Thanks!

[christoph93]

Hi @cibero42, thanks for the support.

I have these variables commented out already. My queue settings are as follows:

I could try using reabbitMQ for the queue...

[graham-web]

I opened this ticket just as a prod to implement something preventing the worker container crashing (which doesn't otherwise seem to cause any particular issue here.)

However, just as @christoph93 and @O-Mutt I'm also seeing mautic:campaigns:trigger in the cron container hitting the memory limit, and that has turned out to be the cause of my original "small percentage of messages not being sent".

If it dies halfway through queuing a batch, the remaining contacts from that batch are not picked up on the next run...

• From observing fluctuations in the send totals on a daily newsletter, it does appear the likelihood of hitting the memory limit and losing messages is proportional to the length of the e-mail going out... i.e. short e-mails send to all contacts no problem. Longer e-mails lose up to 100 contacts. Some e-mails crash twice while sending to 3000 and are more like 200 contacts lost.

• I tried doing some memory profiling of the campaign trigger process. My suspicion is on the string replacement in MailHelper::searchReplaceTokens as the source of the leak, but not experienced in leak-hunting and the intricacies of PHP's garbage collection, so a little unsure how to prove it.

• For now, I've increased the PHP memory limit for the cron container, and will be limiting the number of contacts queued per cron invocation via the --campaign-limit flag to try and ensure no e-mails get dropped on the floor.

[christoph93]

@graham-web Have you had any luck with setting limits?

I've set --batch-size=10 on all cron jobs, I think it helped, but still see this:

mautic-cron | Warning: Failed to set memory limit to 0 bytes (Current memory usage is 2097152 bytes) in Unknown on line 0

Also, I ran all the commands from my crontab manually in the cron container once and never saw a memory limit warning.