GrahamDumpleton
Fatal Python error: Could not allocate TLS entry immediately after graceful restart
Environment & Config Summary
- OS: Ubuntu
- Apache: 2.4.29
- mod_wsgi: 4.5.17
- Python: 3.6
- Mode: WSGIDaemonProcess test python-path=/opt/check/tool user=www-data WSGIProcessGroup test WSGIScriptAlias / /opt/check/tool/test/wsgi.py WSGIPassAuthorization On
Problem Description & Sequence of Events
The server runs normally for a period. The problem is triggered specifically by an Apache graceful restart by logrotate. The server has two logrotate trigger about apache2, Job 1 (Daily Trigger): A log file is rotated daily, and its postrotate script runs invoke-rc.d apache2 reload. Job 2 (Size Trigger): Another log file is rotated when it hits a size limit (which happens about once per day), and its postrotate script runs systemctl reload apache2. (apache2 service - ExecReload=/usr/sbin/apachectl graceful)
I have experienced this issue twice. On both occasions, the problem was preceded by two separate logrotate jobs triggering an Apache reload at almost the same time. While this double-reload appears to be the trigger, we've observed the same event happening frequently in the past without leading to this failure. So I haven't isolated the root cause yet, but this part looks highly suspicious.
log
[Tue Jul 29 00:17:01.555656 2025] [mpm_prefork:notice] [pid 1770] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
[Tue Jul 29 00:17:01.555676 2025] [core:notice] [pid 1770] AH00094: Command line: '/usr/sbin/apache2 -f /etc/opt/httpd_console.conf
[Tue Jul 29 00:17:06.299733 2025] [mpm_prefork:notice] [pid 1770] AH00171: Graceful restart requested, doing restart... (PHP and SSL warnings omitted for brevity) ...
[Tue Jul 29 00:17:09.344299 2025] [mpm_prefork:notice] [pid 1770] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
[Tue Jul 29 00:17:09.344313 2025] [core:notice] [pid 1770] AH00094: Command line: '/usr/sbin/apache2 -f /etc/opt/httpd_console.conf
Fatal Python error: Could not allocate TLS entry
Fatal Python error: Could not allocate TLS entry
Fatal Python error: Could not allocate TLS entry
Fatal Python error: Could not allocate TLS entry
[Tue Jul 29 00:17:10.348963 2025] [core:notice] [pid 1770] AH00052: child pid 8056 exit signal Aborted (6)
Fatal Python error: Could not allocate TLS entry
[Tue Jul 29 00:17:10.349738 2025] [core:notice] [pid 1770] AH00052: child pid 8117 exit signal Aborted (6)
[Tue Jul 29 00:17:10.349751 2025] [core:error] [pid 1770] AH00546: no record of generation 1015 of exiting child 8117
[Tue Jul 29 00:17:10.349759 2025] [core:notice] [pid 1770] AH00052: child pid 8118 exit signal Aborted (6)
Any insight or further debugging steps you could suggest would be greatly appreciated.
Thank you.
Are you really stuck with using such an old Python and mod_wsgi version?
Anyway, try sticking this in the Apache configuration file (outside of any VirtualHost).
WSGIDestroyInterpreter Off
I can't clearly tell from logs you are giving whether error is occurring to processes which are trying to shutdown, or the newly created processes.
That directive may avoid issues if is on destruction of Python interpreter, which can be an issue with some third party Python packages.
If that is the only WSGI application being hosted on the server (or only one per mod_wsgi daemon process group), also try adding:
WSGIApplicationGroup %{GLOBAL}
Some third party packages don't work properly in Python sub interpreters. Avoiding interpreter destruction is one way of avoiding problems with those, as is setting WSGIApplicationGroup to force use of main interpreter context rather than a sub interpreter.
Thanks for the quick response. I will try adding the directives you recommended.
By the way, I was wondering if upgrading the Python and mod_wsgi versions could potentially solve the problem. The product is already deployed, and I think i can't upgrade the Python version. However, I can upgrade the version of the mod_wsgi module.
I have a follow-up question. Since the error first occurred, the WSGI processes have been dying immediately in a repeating cycle (even after the logs I sent). Because of this, I naturally assumed the problem was happening at startup.
However, I believe you mentioned that it was difficult to determine whether the error was happening at startup or shutdown. This brings me to my question: Does an error that occurs during one process's shutdown have any effect on the initialization of the next process?
Thank you.
The shutdown of one process should not affect the next to be created.
Is the site handling any requests at all? If it is working okay otherwise and you only see this error when log rotate happens, then it is likely only occurring on shutdown. Using info logging level will help confirm.
I'm reporting the results after a test.
First, regardless of the interval I used for a graceful restart of Apache2, a problem occurred around the 1017th attempt. For this, I was using mod_wsgi 4.9.4 and had added the WSGIDestroyInterpreter Off directive, but the same problem occurred.
Finally, even when I tested with only the LoadModule directive remaining, the problem arose after a similar number of repetitions.
Could this possibly be an issue with Python 3.6?
Thank you.
+) After posting this question, I upgraded to Python 3.11 and ran it, but the same issue occurs. This time, it leaves a similar log, but in a slightly different format.
[Wed Aug 13 17:07:23.568648 2025] [wsgi:info] [pid 27511] mod_wsgi (pid=27511): Initializing Python.
Fatal Python error: _PyGILState_Init: memory allocation failed
Python runtime state: preinitialized
A bit of an explanation about how Apache loads modules and when Python is initialised.
When Apache starts up it starts as the root user. This is what is called the main Apache parent process. All Apache modules are loaded in the context of this parent process.
From the Apache parent processes, new processes fork. This is done for the Apache child worker processes, as well as the mod_wsgi daemon processes.
Immediately after forking this new processes, privileges are dropped and code then runs as the Apache user and not the root user.
At this point any Apache modules have the chance to run additional initialisation in the child processes.
In the case of mod_wsgi, it is only at this point that the Python interpreter is initialised and started.
When an Apache reload or graceful restart is done, any forked child process are signalled to shutdown.
At the same time, in the Apache parent process, any Apache modules are potentially unloaded from memory with hooks called first so modules have the opportunity to perform cleanup actions.
The Apache modules are then reloaded and reinitialised, in the same context of the parent Apache process.
New child processes are forked and same steps as noted above occur.
In the case of mod_wsgi, the Python interpreter is only initialised in the child processes. As such, anything the Python interpreter does, or your application, cannot affect the operation of future forked processes, as forked processes always inherited the memory state of the Apache parent process only.
What this means is that a failure like you are seeing, which looks like memory corruption, or less like memory exhaustion, can only occur due to bugs/memory leaks in the initialising and unloading code of any Apache modules.
Such bugs/memory leaks, although affecting mod_wsgi may have nothing to do with mod_wsgi.
A major culprit could actually be the PHP module if you are also loading this in the Apache instance, as its execution model means that it initialises PHP in the Apache parent process, not in the child processes like what happens with Python in mod_wsgi.
Thus there is a much bigger risk that PHP can cause issues if it is also being used.
So what is the set of Apache modules you are loading in the Apache instance. Are there other complicated Apache modules being used such as PHP, mod_perl or others?
Also, if you are only using mod_wsgi daemon mode, add to your Apache configuration (outside of any VirtualHost), the directive:
WSGIRestrictEmbedded On
This will ensure that Python is not initialised in Apache child worker processes, and only initialises Python in the mod_wsgi daemon processes. In other words, it ensures Python is not initialised in embedded mode, since looks like you are not using that.
This strictly should not make any difference with the behaviour you are seeing, because as I said this only happens in the child process anyway, and what looks to be happening is that your child processes are inheriting memory corruption issues from the parent process.
Anyway, what Apache modules do you actually have enabled and loaded into Apache?
First, I've copied the list of modules I'm using as is:
LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule php7_module modules/libphp7.2.so <- php
LoadModule headers_module modules/mod_headers.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule wsgi_module modules/mod_wsgi.so <- wsgi
What makes this more challenging is that the problem does not occur when I run the aforementioned test with either only mod_wsgi enabled (after removing the PHP-related module) or with mod_wsgi removed.
Could there be a possibility of a memory-related error occurring when both modules are used simultaneously?
But do you use PHP? If you don't then don't enable it.
Also, are you configuring Apache to terminate SSL connections such that your site is accessed using https.
There have been cases in the past where the operating system had updated SSL libraries which were slightly different compared to what Python was compiled for. This would caused strange SSL problems and in some cases crashes as well.
If you aren't relying on Apache SSL module, also disable that.