benoitc
setproctitle does not work together with systemd socket support
Using gunicorn 23.0.0 on Linux with systemd service+socket unit setup similar to the one described in docs, I found that the setproctitle support does not work (ps still shows full command lines). After adding SPT_DEBUG=1 to the environment, I see the following in the errorlog file:
[2025-10-21 16:14:46 +0300] [8803] [INFO] Starting gunicorn 23.0.0
[2025-10-21 16:14:46 +0300] [8803] [INFO] Listening at: unix:/run/REDACTED.sock (8803)
[2025-10-21 16:14:46 +0300] [8803] [INFO] Using worker: sync
[SPT]: reading argc/argv from Python main
[SPT]: found 4 arguments
[SPT]: walking from environ to look for the arguments
[SPT]: found environ at 0x7ffc84a43d16
[SPT]: found argv[3] at 0x7ffc84a43d09: LISTEN_FDS=1
[SPT]: found argv[2] at 0x7ffc84a43cdc: /REDACTED/gunicorn.conf.py
[SPT]: found argv[1] at 0x7ffc84a43cd9: -c
[SPT]: argv[0] should be at 0x7ffc84a43cc8
[SPT]: argv[0] '/REDACTED/gunicorn' doesn't match '/usr/bin/python3'
[SPT]: couldn't find argv from environ
[SPT]: get_argc_argv failed
[SPT]: failed to initialize setproctitle
Apparently setproctitle starts looking for argv in a wrong place (it thinks that LISTEN_FDS=1 is the last item in argv when it's actually the first item in environ passed by systemd), therefore it does not find a proper argv[0] and fails.
The problem is probably caused by the environment manipulation in gunicorn.systemd.listen_fds() which happens before the first setproctitle() call — popping the first item in environ probably changes the environ pointer at the C level, breaking the setproctitle init code which runs at the first call and searches for argv[0] backwards from environ. So it looks like all environ changes before the first call to setproctitle() need to be avoided.
I have been able to find a workaround — adding the following piece of code to gunicorn.conf.py makes setproctitle work again:
try:
from setproctitle import setproctitle
setproctitle("gunicorn: init")
except Exception:
pass
In this case the initialization of setproctitle happens before the environ change, and subsequent calls to setproctitle() can work properly. The SPT_DEBUG=1 info now gets captured in the systemd journal (timestamp and hostname fields trimmed):
systemd[1]: Starting [email protected] - gunicorn@REDACTED daemon...
start-gunicorn[8858]: [SPT]: module init
start-gunicorn[8858]: [SPT]: reading argc/argv from Python main
start-gunicorn[8858]: [SPT]: found 4 arguments
start-gunicorn[8858]: [SPT]: walking from environ to look for the arguments
start-gunicorn[8858]: [SPT]: found environ at 0x7fff79b6cd09
start-gunicorn[8858]: [SPT]: found argv[3] at 0x7fff79b6ccdc: /REDACTED/gunicorn.conf.py
start-gunicorn[8858]: [SPT]: found argv[2] at 0x7fff79b6ccd9: -c
start-gunicorn[8858]: [SPT]: found argv[1] at 0x7fff79b6ccc7: /REDACTED/gunicorn
start-gunicorn[8858]: [SPT]: argv[0] should be at 0x7fff79b6ccb6
start-gunicorn[8858]: [SPT]: found argv[0]: /usr/bin/python3
systemd[1]: Started [email protected] - gunicorn@REDACTED daemon.
Should gunicorn itself have a workaround for this problem (calling setproctitle() early enough, so that it has a chance to init properly)?
Unable to reproduce on Linux 6.1, setproctitle 1.3.7, gunicorn master. process title is overwritten just fine for me. This is what I usually do:
WorkingDirectory=~
ExecStart=@/usr/bin/python3 bin/python3 -m gunicorn --log-level=debug -- mywsgi:application
This also worked for me:
ExecStart=/home/mywsgi/bin/python3 /home/mywsgi/bin/gunicorn --log-level=debug -- mywsgi:application
How does your .service file look like, what else does your application/config do at import time? Any other C imports or wrapper scripts?
I can reproduce the problem even with this dummy.py:
def application(environ, start_response):
status = '200 OK'
headers = [('Content-type', 'text/plain; charset=utf-8')]
start_response(status, headers)
return [b"Hello, World!"]
and the following commands:
python3 -m venv venv
. ./venv/bin/activate
pip install gunicorn setproctitle
sh -c 'SPT_DEBUG=1 LISTEN_PID=$$ LISTEN_FDS=0 exec gunicorn dummy:application'
The last command emulates the file descriptor passing protocol used by systemd, but specifies that 0 descriptors are passed, therefore gunicorn reverts to the traditional listen behavior, but still executes the code which removes LISTEN_FDS and LISTEN_PID from the environment. Also the reproduction may depend on how exactly the shell prepares the environment for the exec'ed process — here bash 5.2.37(1)-release passes LISTEN_FDS as the first environment variable, which is required to reproduce the bug. So maybe a Python reproducer like the following would be more reliable:
python3 -c 'import os; os.execlpe("gunicorn", "gunicorn", "dummy:application", {"LISTEN_FDS":"0","LISTEN_PID":str(os.getpid()),"SPT_DEBUG":"1", **os.environ})'
The output of the last command in my case is:
[SPT]: module init
[2025-10-21 23:08:39 +0300] [15312] [INFO] Starting gunicorn 23.0.0
[2025-10-21 23:08:39 +0300] [15312] [INFO] Listening at: http://127.0.0.1:8000 (15312)
[2025-10-21 23:08:39 +0300] [15312] [INFO] Using worker: sync
[SPT]: reading argc/argv from Python main
[SPT]: found 3 arguments
[SPT]: walking from environ to look for the arguments
[SPT]: found environ at 0x7fff64d6451c
[SPT]: found argv[2] at 0x7fff64d6450f: LISTEN_FDS=0
[SPT]: found argv[1] at 0x7fff64d644fd: dummy:application
[SPT]: argv[0] should be at 0x7fff64d644d0
[SPT]: argv[0] 'REDACTED/venv/bin/gunicorn' doesn't match 'REDACTED/venv/bin/python3'
[SPT]: couldn't find argv from environ
[SPT]: get_argc_argv failed
[SPT]: failed to initialize setproctitle
[SPT]: setup was called more than once!
[SPT]: failed to initialize setproctitle
[2025-10-21 23:08:39 +0300] [15313] [INFO] Booting worker with pid: 15313
@pajod Do you actually use a .socket unit in your configuration? The problem is triggered by systemd passing the socket descriptors to the gunicorn process using the LISTEN_PID and LISTEN_FDS environment variables; it probably won't appear if you are specifying listening sockets in some other way.
Also this issue is even noted in the setproctitle documentation:
You should import and use the module (even just calling
getproctitle()) pretty early in your program lifetime: code writing env vars may interfere with the module initialisation.
Actually it turns out that systemd passes environment variables in a different order that probably avoids the bug:
LANG=ru_RU.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
NOTIFY_SOCKET=/run/systemd/notify
LISTEN_PID=11334
LISTEN_FDS=1
LISTEN_FDNAMES=...
So if my systemd service would start gunicorn directly, the bug would probably be hidden. However, I actually start a wrapper script like this (so that the same service could be used even if using system packages instead of venv):
#!/bin/sh
set -efu
dir="${0%/*}"
deployment_dir="$(realpath -e "${dir}/..")"
venv_script="${deployment_dir}/venv/bin/activate"
if [ -e "${venv_script}" ]; then
. "${venv_script}"
fi
exec gunicorn "$@"
And that invocation of /bin/sh happens to change the order of environment variables — LISTEN_FDS=1 becomes first, and the bug shows up. (This happens even when the only command in the script is exec <something> without any attempts to change anything in the environment.)
Anyway, given the fact that setproctitle documentation says that the environment must not be changed before its initialization, maybe changing the gunicorn code to respect that restriction would still make sense.
Is LISTEN_FDS the first thing in your inherited environment? How.. do you even do that, with systemd?
edit: I see. So for now, the easiest workaround is probably to add a dummy env variable. This works for me:
python3 -c 'import os; os.execlpe("gunicorn", "gunicorn", "dummy:application", {"LIKE_SYSTEMD":"", "LISTEN_FDS":"0","LISTEN_PID":str(os.getpid()),"SPT_DEBUG":"1", **os.environ})'
(You don't need that bin/activate thing for regular virtual environment setup. Use the ExecStart=@ form to let systemd pass a suitable argv[0] and cPython will know what to do.)
About fixing this: setproctitle needs to be setup early so that we can avoid confusing its detection before-fork (because not polluting worker process environ seems like the right thing to do), yet at the same time setproctitle must be setup late on modern-ish Mac because the library it loads there refuses to work properly after-fork, see #3021. I have no immediate ideas on an elegant solution about when to change env, and when to setup setproctitle.