GrahamDumpleton
After check_password is invoked, apache process is unable to terminate by SIGTERM.
After check_password is invoked, apache process is unable to terminate by SIGTERM (and is killed by SIGKILL). When check_password is not invoked, apache process is terminated by SIGTERM normally.
It occurs on mod-wsgi 5.0.0 and 5.0.1dev1 (develop branch) with Python 3.10.14.
error.log
[Thu Aug 08 21:37:54.433148 2024] [mpm_worker:notice] [pid 3304855:tid 140151734205504] AH00292: Apache/2.4.41 (Ubuntu) mod_wsgi/5.0.0 Python/3.10 SVN/1.13.0 configured -- resuming normal operations
[Thu Aug 08 21:37:54.433250 2024] [core:notice] [pid 3304855:tid 140151734205504] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND -f /home/jun66j5/src/trac/modwsgi.conf'
[Thu Aug 08 21:38:37.667061 2024] [core:warn] [pid 3304855:tid 140151734205504] AH00045: child process 3304858 still did not exit, sending a SIGTERM
[Thu Aug 08 21:38:39.669145 2024] [core:warn] [pid 3304855:tid 140151734205504] AH00045: child process 3304858 still did not exit, sending a SIGTERM
[Thu Aug 08 21:38:41.670293 2024] [core:warn] [pid 3304855:tid 140151734205504] AH00045: child process 3304858 still did not exit, sending a SIGTERM
[Thu Aug 08 21:38:43.672387 2024] [core:error] [pid 3304855:tid 140151734205504] AH00046: child process 3304858 still did not exit, sending a SIGKILL
[Thu Aug 08 21:38:44.673507 2024] [mpm_worker:notice] [pid 3304855:tid 140151734205504] AH00295: caught SIGTERM, shutting down
access.log
127.0.0.1 - jun66j5 [08/Aug/2024:21:38:01 +0900] "GET /svn/ HTTP/1.1" 200 535
ps xf
651297 ? S 0:34 sshd: jun66j5@pts/0
651298 pts/0 Ss 0:04 \_ -bash
3304842 pts/0 S+ 0:00 \_ /bin/sh /home/jun66j5/bin/trac.sh modwsgi /dev/shm/trac16 /dev/shm/tracenv
3304855 pts/0 S+ 0:00 \_ /usr/sbin/apache2 -DFOREGROUND -f /home/jun66j5/src/trac/modwsgi.conf
3304856 pts/0 Sl+ 0:00 \_ (wsgi:trac) -DFOREGROUND -f /home/jun66j5/src/trac/modwsgi.conf
3304857 pts/0 Sl+ 0:00 \_ (wsgi:trac) -DFOREGROUND -f /home/jun66j5/src/trac/modwsgi.conf
3304858 pts/0 Sl+ 0:00 \_ /usr/sbin/apache2 -DFOREGROUND -f /home/jun66j5/src/trac/modwsgi.conf
3304859 pts/0 Sl+ 0:00 \_ /usr/sbin/apache2 -DFOREGROUND -f /home/jun66j5/src/trac/modwsgi.conf
Part of modwsgi.conf
<VirtualHost *:3000>
ServerAdmin webmaster@localhost
DocumentRoot ${_TMPDIR}/docroot
SetEnv trac.env_parent_dir ${_TMPDIR}/trac
WSGIDaemonProcess trac \
python-home=${_VENVDIR} display-name=%{GROUP} \
processes=2 threads=25 maximum-requests=128 inactivity-timeout=600
WSGIScriptAlias / ${_TMPDIR}/trac.wsgi \
process-group=trac application-group=%{GLOBAL}
Alias /svn ${_TMPDIR}/svn
<Location />
AuthType Basic
AuthBasicProvider wsgi
WSGIAuthUserScript ${_TMPDIR}/auth.wsgi application-group=auth
AuthName "auth"
Require all granted
</Location>
<Location /*/login>
Require valid-user
</Location>
<Location /*/login/*>
Require valid-user
</Location>
<Location /svn>
SVNAdvertiseV2Protocol On
DAV svn
SVNPath ${_TMPDIR}/svn
AuthzSVNAnonymous On
#AuthzSVNNoAuthWhenAnonymousAllowed On
AuthzSVNAccessFile ${_PWD}svn-authz
Satisfy Any
Require valid-user
</Location>
</VirtualHost>
auth.wsgi
def check_password(environ, user, password):
return True
What happens if you add:
WSGIDestroyInterpreter Off
at global Apache scope?
Ok. After adding WSGIDestroyInterpreter Off, apache process is terminated by SIGTERM normally.
So that you saw this when check_password() was used is I believe coincidental.
The underlying issue here and which have seen with Trac before, is that in Python 3.9 the behaviour of Python interpreter shutdown in core Python code was changed to require that background threads be stopped before the process could be stopped. If they weren't the process would hang waiting for them.
The consequence of this is if a web application under mod_wsgi created background threads and doesn't use Python atexit module hook, or mod_wsgi's own shutdown hook, to try and signal the threads to somehow stop, then process would hang on shutdown.
The solution to this was to add that WSGIDestroyInterpreter directive so that mod_wsgi could be told to not attempt to do a graceful cleanup of the Python interpreter on shutdown and instead just let the process exit.
There are more details on this issue in changes notes for version added:
- https://modwsgi.readthedocs.io/en/master/release-notes/version-4.9.1.html
Hm, I confirmed that apache is terminated normally even if check_password() is invoked after removing WSGIDaemonProcess trac from the configuration.
BTW, the WSGIDestroyInterpreter directive seems to be not listed at https://modwsgi.readthedocs.io/en/master/configuration.html
Going to close this issue now. I have added some docs about WSGIDestroyInterpreter in develop version of code for next version.