killling workers takes too much time

[ryanking8215]

My environment is Flask app and a user thread to listen the redis's message. I used uwsgi to drive the app. But when I press ctrl-c to quit the uwsgi, uwsgi will take long time and say:

workers … is taking too much time to die…NO MERCY

If I disable the user thread , everything is ok.

I tried many ways to resolve it.

signal
uwsgi.signal
uwsgi.atexit

But they all don't work.

here is my uwsgi command

uwsgi --http-socket 127.0.0.1:8800 --virtualenv=/opt/python_venv --wsgi-file /opt/app.py --callable app --processes 4 --threads 2 --stats 127.0.0.1:9191

Thanks for any suggestion！

[unbit]

Destroying threads in the POSIX world is basically no-way :) The right approach is letting your thread return when the destroy procedure is triggered. You have a bunch of ways to do it, but personally i am way more brutal and i generally set --worker-reload-mercy to a couple of seconds :)

[ryanking8215]

In fact, I use uwsgi.atexit to get the destroy signal successfully , and in the callback I call the function which make redis listening thread quit. But worker still can't be killed until timeout.

[unbit]

How do you 'quit' the thread ?

[ryanking8215]

I wrote a simple demo, the problem is I couldn't get the destroy signal.

from flask import Flask
import threading
import time

app=Flask(__name__)

@app.route('/')
def index():
    return 'hello,world'

a=1
def myjob():
    print("my job run",a)
    while a!=0:
        time.sleep(1)
    print(">>>>>>>> my job quit")

thrd = threading.Thread(target=myjob)
thrd.start()

def bye():
    print(">>>>>>>>>>> bye")
    a=0

try:
    import uwsgi
    uwsgi.atexit = bye
except:
    pass

import signal
signal.signal(signal.SIGINT,bye)
signal.signal(signal.SIGQUIT,bye)

if __name__=='__main__':
    app.run()

Neither uwsgi.atext nor signal could get the destroy signal. Unless the thread is not started.

[anthonyrisinger]

You did not enable app threads with --enable-threads, probably you are having GIL issues.

Also, you cannot catch signals in python without a specific option enabled that I forget offhand, possibly only available in 2.1.

In general, you should avoid --threads for python apps... they cannot be executed concurrently because of GIL. That option creates C-level threads (unmanaged by python) to run your WSGI app in.

edit: also, your bye function will not work because a is a global; you need to add global a to the top of the function.

[ryanking8215]

You are right about global a stuff. -:) I know the threads is not the proper way in python because of GIL. -:)

But the problem is when I pressed 'Ctrl-c' to quit the uwsgi, bye() function was not called. I don't know how to make it called to quit the thread

[xrmx]

@ryankask is this still an issue? btw if you are doing pubsub the redis-py's pubsub object has a run_in_thread method to help you with this.

[Vingtoft]

Did anyone find a solution to this problem? I have the exact same issue.

[Kronuz]

Same problem here... uwsgi.atexit didn't work and neither using signal.signal nor uwsgi.signal Is there a bug or this just can't be done using uwsgi.

[unbit]

First of all, destroying threads is not possible on POSIX systems without thread cooperation itself, unfortunately there is no easy-solution. If you want to rely on UNIX signals (and very probably you are opening a new can of worms, but they should at least work 99% of the times) you have to explicitely tell the python VM you want to override signals management using the --py-call-osafterfork option.

[autumnjolitz]

I've learned today that if my flask app on uWSGI uses enable-threads=true and threads=2 (for any number greater than 1), then a SIGHUP on a worker produces an indefinite hang.

[zsluedem]

i meet this kind of issue too.My config threads=2. When i try to kill the uwsgi byuwsgi --stop uwsgi.pid, it wait for a while to kill and show the log take too much time. And when i try to realod the uwsgi , the uwsgi can't accept any request any more. It just hang there and do nothing.But sometimes it worked, it is kind of mysterious. My app is a python app.And i found this and it seems python doesn't support threaded.

I will try to use the app without threaded for a while and see how.

[erny]

Could this be similar to my issue #1599 ?

[yamyamyuo]

reload-mercy = int

worker-reload-mercy = int

add these two config works for me. https://stackoverflow.com/questions/37846202/nginx-python-uwsgi-kill-issue

[bernd-wechner]

reload-mercy = int

worker-reload-mercy = int

Alas didn't work for me. Tried it and still painfully slow to stop uwsgi. Be nice if it offered a verbose log on the stop so we could check once down what took that time). I do reloads mostly as they are fast and stop/start so painful. Alas I have to stops to some database maintenance ops.

[bernd-wechner]

Would be awfully nice to be able to enable some kind of debug logging to watch a kill trace and see where the time is going.