unbit
echo f > /opt/foo/running_instance.fifo stuck
Hi,
I'm using the master fork process as described at http://zarnovican.github.io/2016/02/15/uwsgi-graceful-reload/ to gracefully reload.
Most of the time it works fine, but once in a while the deployment gets stuck while
echo f > /opt/foo/running_instance.fifo
On further debugging, it looks like the inode for the fifo file has changed.
lsof|grep fifo
uwsgi 17347 ubuntu 21r FIFO 202,1 0t0 1191611 /opt/foo/running_instance.fifo (deleted)
ls -i /opt/foo/running_instance.fifo
1193072 /opt/foo/running_instance.fifo
Here's a snippet of the uwsgi.ini file:
master = true
master-fifo = /opt/foo/new_instance.fifo
master-fifo = /opt/foo/running_instance.fifo
hook-accepting1-once = write:<%= @app_directory %>/hook.ready ok
hook-as-user-atexit = unlink:<%= @app_directory %>/hook.ready
if-exists = /opt/foo/running_instance.fifo
hook-accepting1-once = writefifo:/opt/foo/running_instance.fifo q
endif =
hook-accepting1-once = spinningfifo:/opt/foo/new_instance.fifo 1P
processes = 5
threads = 1
socket = /opt/foo/nginx.sock
chmod-socket = 666
vacuum = false
enable-threads = true
single-interpreter = true
pidfile = /opt/foo/main.pid
Can someone help me figure out why that would happen?
-Saurav
@xrmx uwsgi = 2.0.15 ubuntu = 14.04
Seems like I am running into https://github.com/unbit/uwsgi/issues/1492
I have encountered this very issue before, I wrote an article about it: https://messense.me/uwsgi-re-fork-master
(It's in Chinese but you should be able to Google translate it I think)
The basic idea is switching to a different master-fifo when quitting.
[uwsgi]
chdir = /home/foo/helloworld
module = helloworld.wsgi
processes = 4
socket = /var/run/uwsgi/app.sock
pidfile = /var/run/uwsgi/app.pid
vacuum = false
master-fifo = /tmp/app.new.fifo
master-fifo = /tmp/app.running.fifo
master-fifo = /tmp/app.quit.fifo
if-exists = /tmp/app.running.fifo
hook-accepting1-once = writefifo:/tmp/app.running.fifo 2q
endif =
hook-accepting1-once = spinningfifo:/tmp/app.new.fifo 1P
I have encountered this very issue before, I wrote an article about it: https://messense.me/uwsgi-re-fork-master
(It's in Chinese but you should be able to Google translate it I think)
The basic idea is switching to a different master-fifo when quitting.
[uwsgi] chdir = /home/foo/helloworld module = helloworld.wsgi processes = 4 socket = /var/run/uwsgi/app.sock pidfile = /var/run/uwsgi/app.pid vacuum = false
master-fifo = /tmp/app.new.fifo master-fifo = /tmp/app.running.fifo master-fifo = /tmp/app.quit.fifo
if-exists = /tmp/app.running.fifo hook-accepting1-once = writefifo:/tmp/app.running.fifo 2q endif = hook-accepting1-once = spinningfifo:/tmp/app.new.fifo 1P
Hi really thank you for this method. We used it many years. But now after uwsgi 2.0.27 it's - broken. The old master is stuck in memory after forking the new one. It tries to kill the old workers (but these workers are already gracefully closed):
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 72531)
sendmsg(): Transport endpoint is already connected [core/notify.c line 39]
spawned uWSGI worker 1 (pid: 72533, cores: 1)
running "write:/tmp/online_ex-2503251348.ready ok" (accepting1-once)...
running "writefifo:/tmp/running-online_ex.fifo 2q" (accepting1-once)...
spawned uWSGI worker 2 (pid: 72534, cores: 1)
*** Stats server enabled on /tmp/stat-online_ex.sock fd: 15 ***
running "spinningfifo:/tmp/new-online_ex.fifo 1P" (master-start)...
waiting for /tmp/new-online_ex.fifo ...
Tue Apr 29 13:45:45 2025 - active master fifo is now /tmp/running-online_ex.fifo
writing pidfile to /tmp/online_ex.pid
Tue Apr 29 13:45:45 2025 - active master fifo is now /tmp/quit-online_ex.fifo
Tue Apr 29 13:45:45 2025 - graceful shutdown triggered...
Gracefully killing worker 2 (pid: 72477)...
Gracefully killing worker 1 (pid: 72476)...
Tue Apr 29 13:46:46 2025 - worker 1 (pid: 72476) is taking too much time to die...NO MERCY !!!
Tue Apr 29 13:46:46 2025 - worker 2 (pid: 72477) is taking too much time to die...NO MERCY !!!
