grisha
Mod Python causing segmentation faults when compiled with Python3.9
Hello,
I have recently upgraded from python3.7 to python3.9 and compiled mod_pyhton with python3.9.
After the upgrade, I see apache process crashing with segmentation fault intemittently.
I generated core dump and following is the traceback which I see.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--l
Core was generated by `/usr/sbin/apache2 -d /var/service_dispatcher -X -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 release_sentinel (wr_raw=0x7fd5ee538810) at ../Modules/_threadmodule.c:1246
1246 ../Modules/_threadmodule.c: No such file or directory.
[Current thread is 1 (Thread 0x7fd5e47f8700 (LWP 62175))]
(gdb) bt
#0 release_sentinel (wr_raw=0x7fd5ee538810) at ../Modules/_threadmodule.c:1246
#1 0x00007fd5f17f0a2a in release_interpreter (idata=0x564a091fcb40) at mod_python.c:306
#2 python_handler (req=0x7fd5c496f0a0, phase=<optimized out>) at mod_python.c:1573
#3 0x0000564a08e0c2d0 in ap_run_fixups ()
#4 0x0000564a08e0ea3b in ap_process_request_internal ()
#5 0x0000564a08e30978 in ap_process_async_request ()
#6 0x0000564a08e30bce in ap_process_request ()
#7 0x0000564a08e2ca44 in ?? ()
#8 0x0000564a08e216e0 in ap_run_process_connection ()
#9 0x00007fd5f18522d3 in ?? () from /usr/lib/apache2/modules/mod_mpm_worker.so
#10 0x00007fd5f1b30ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fd5f1a50aef in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)
The traceback points to following line in mod_python.c,
following is the code block for reference
static void release_interpreter(interpreterdata *idata)
{
PyThreadState *tstate = PyThreadState_Get();
#ifdef WITH_THREAD
PyThreadState_Clear(tstate);
if (idata)
APR_ARRAY_PUSH(idata->tstates, PyThreadState *) = tstate;
else
PyThreadState_Delete(tstate);
PyEval_ReleaseThread(tstate);
#else
if (!idata) PyThreadState_Delete(tstate);
#endif
}
Looking at the code, we first create a PyThreadState object,
then make a call to PyThreadState_Clear(tstate) and if idata is empty, we make call to PyThreadState_Delete(tstate).
but in python3.9, there is a change in behaviour of PyThreadState_Clear and PyThreadState_Delete methods.
If we check the official documenatation of python, it states
void PyThreadState_Clear(PyThreadState *tstate) Part of the Stable ABI. Reset all information in a thread state object. The global interpreter lock must be held.
Changed in version 3.9: This function now calls the PyThreadState.on_delete callback. Previously, that happened in PyThreadState_Delete().
however when PyThreadState.on_delete callback is called, we see segmentation fault.
Following is the pull request which introduced this change,
https://github.com/python/cpython/pull/18296
In the changed files we can see tstate->on_delete hook call is moved from PyThreadState_Delete to PyThreadState_Clear,
In order to avoid call to this hook,
I changed tstate->on_delete to point to null and following is th change which I did in mod_python.c's release_interpreter code block
static void release_interpreter(interpreterdata *idata)
{
PyThreadState *tstate = PyThreadState_Get();
#ifdef WITH_THREAD
tstate->on_delete = NULL;
PyThreadState_Clear(tstate);
if (idata)
APR_ARRAY_PUSH(idata->tstates, PyThreadState *) = tstate;
else
PyThreadState_Delete(tstate);
PyEval_ReleaseThread(tstate);
#else
if (!idata) PyThreadState_Delete(tstate);
#endif
}
After this change, mod_python and apache is working completely fine, However I am not sure if this is right fix or not.
Let me know if there are other ways to fix this issue and make mod_python compatible with python3.9
Otherwise I can raise a Pull request with that change.
Attaching core dump.
Thank You
@rohan-97 Thanks for looking into this.
I think you'll need to dig deeper - setting tstate->on_delete to NULL prevents whatever it was from being executed, but it was there for a reason.
The traceback from gdb points to ../Modules/_threadmodule.c:1246 - which looks like it's a comment, at least looking at the 3.9 tag:
https://github.com/python/cpython/blob/3.9/Modules/_threadmodule.c#L1246
...which doesn't make sense, most likely your Python is from a slightly different version, perhaps patched by the distribution or something like that - you might want to try to track down exactly which line of code it is that causes the segfault.
