segfault when releasing a thread

[wom-bat]

I see apparently at random, a segmentation violation. Bcktrace is

(gdb) bt
#0  raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  <signal handler called>
#2  __new_sem_post (sem=0x21) at sem_post.c:36
#3  0x00007f38785080f9 in PyThread_release_lock ()
   from /lib/x86_64-linux-gnu/libpython3.9.so.1.0
#4  0x00007f3878570be9 in ?? () from /lib/x86_64-linux-gnu/libpython3.9.so.1.0
#5  0x00007f38788d4a2a in release_interpreter (idata=0x55da959531c0)
    at mod_python.c:306
#6  python_handler (req=0x7f387450d0a0, phase=<optimized out>)
    at mod_python.c:1573
#7  0x000055da949c4bb0 in ap_run_handler ()
#8  0x000055da949c51a6 in ap_invoke_handler ()
#9  0x000055da949ddabb in ap_process_async_request ()
#10 0x000055da949ddcfe in ap_process_request ()
#11 0x000055da949d9bad in ?? ()
#12 0x000055da949ce860 in ap_run_process_connection ()
#13 0x00007f3878960133 in ?? () from /usr/lib/apache2/modules/mod_mpm_worker.so
#14 0x00007f3878c5eea7 in start_thread (arg=<optimized out>)
    at pthread_create.c:477
#15 0x00007f3878b8edef in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Looks like we're trying to release a semaphore we don't hold.

Python 3.9.1 (Debian build), Apache2.4.46-4 (Debian); mod_python from 88e444429ec with the libpython fix.

Using mpm_worker if that makes a difference.

[wom-bat]

More debugging --- loaded libpython-3.9d.so with debugging syms; the bug is because we're trying to release the curent thread.

[grisha]

I'm not set up to test/verify any of this so if you get to the bottom of it it'd be fantastic. It's (at first glance) code that stood the test of time, so I wonder if there is something about Python 3.9.1 - perhaps it's doing something differently?

[cryptofunallday]

Also been getting this for several months, basically since python went to 3.9 it has been happening. Using version 3.8 seems to make it go away, not really best practices to do that though.

[wom-bat]

Commenting out the call to PyThreadState_Clear(tstate); near line 306 of mod_python.c makes the issue go away. There is only one documented change for this function: from Python 3.9 on it calls the PyThreadState.on_delete callback.

[domruf]

We use httpd docker images which recently switched from debian buster to bullseye. I now also see the Segmentation fault errors with my PythonAuthenHandler. It looks like the errors happen mainly right after a restart of httpd. Is there anything I can do to prevent these errors?

[i300220]

This opens a python file I've been running since python 3.9 at least 2 years ago. The file is called several times per second 24/7. I'm now running python 3.11 and is segfaults horribly. Commenting out PyThreadState_Clear(tstate); fixed it.

Just a confirmation that wom-bat patch works.

[grisha]

So should that line be wrapped in something like so

#if PY_MAJOR_VERSION == 3  && PY_MINOR_VERSION < 10 
    PyThreadState_Clear(tstate);
#endif

[i300220]

So should that line be wrapped in something like so

#if PY_MAJOR_VERSION == 3  && PY_MINOR_VERSION < 10 
    PyThreadState_Clear(tstate);
#endif

Totally makes sense. Thanks!

[grisha]

Could you confirm that this works for you? I'm currently not set up to test it. If it works, then a pull-request would be much appreciated, if not let me know and I'll add it in myself. Thank you!

[i300220]

Could you confirm that this works for you? I'm currently not set up to test it. If it works, then a pull-request would be much appreciated, if not let me know and I'll add it in myself. Thank you!

It does. Just recompiled the source from debian 12 with your suggestion. Restarted apache and no segfault.

Can't make a pull request though, permission denied: $ LANG=C git push origin mb remote: Permission to grisha/mod_python.git denied to i300220. fatal: unable to access 'https://github.com/grisha/mod_python.git/': The requested URL returned error: 403

[grisha]

That's okay, I took care of it, thanks for your help!

[stefanor]

This was fixed as >= 3.10, but reported against 3.9 too. Is that related, or is #122 a different bug

[grisha]

@stefanor I don't have the set up to test it - if you can confirm that the fix should include 3.9 too that'd be much appreciated.

[i300220]

I'm afraid it's present in 3.9 too. I didn't realised it immediately. Only when I installed debian 12 and after many useless edits in my python files because I had a lot of trouble and noticed this fix applied to 3.11 has nothing to do with my python files being bad because the bug was suddenly gone. It was related to this bug. Sorry I forgot to report it. I remember clearly I had to switch to old apache module whenever I wanted to run some python 3 code.

In other words, it was fine on debian 10, broken in debian 11 (with 3.9) and fine again in debian 12, thanks to this fix.

[grisha]

@i300220 so the fix should really be

#if PY_MAJOR_VERSION == 3  && PY_MINOR_VERSION < 9 
   PyThreadState_Clear(tstate);
#endif

does this look reasonable?

[i300220]

@grisha that's right. thanks!