Issue with Gunicorn server or thread hanging when ran from systemd service

[m57]

I apologise in advance for the wording of this question...

I have this very weird issue whereby gunicorn process hangs after I perform a task. The hanging only occurs when the gunicorn process is ran with systemd as a service, but not when ran from the command line.

Ubuntu 18.04.1 Gunicorn Version: 19.9.0 ansible==2.7.5

The flow is:

server.py

import eventlet
eventlet.sleep()
eventlet.patcher.monkey_patch(select=False, socket=True)

from flask import Flask
from flask_socketio import SocketIO, join_room
from flask_assets import Environment
from flask_login import current_user
...
app = Flask(__name__)
app.logger.addHandler(handler)
app.logger.setLevel(logging.INFO)

assets = Environment(app)
assets.register(bundles)

socketio_app = SocketIO(app, logger=False, engineio_logger=False, message_queue="redis://")

From the Web UI user tries to deploy ansible playbook to server:

 ...
                        pbex = PlaybookExecutor(playbooks=[playbook_path], inventory=inventory, variable_manager=variable_man$
                        result = pbex.run()
                        return result

I think ansible here uses subprocess to spawn a command line instance of ansible-playbook but not 100% sure...

result eventually returned to:

                @socketio.on('deploy_ansible_playbook')
                @login_required
                @log_request
                @check_permission('XXX')
                def deploy_ansible_playbook(data):
                        try:  
                                ....
                                result = cur.deploy_ansible_playbook(data)
                                cur.close()
                                ## alert result
                                socketio.emit('alert_msg', result, room=current_user.username)
                        except Exception as e:
                                current_app.logger.error(str(e))
                                socketio.emit('alert_msg', {"type":"warning","data":str(e)}, room=current_user.username)

Issue arises when gunicorn is ran from systemd:

[Unit]
Description=Gunicorn Server

[Service]
Restart=on-failure
WorkingDirectory=/var/www/xxxxx
User=xxxx
ExecStart=/usr/bin/gunicorn architect:app --bind 127.0.0.1:6060 --workers 3 --worker-class eventlet
ExecReload=/bin/kill HUP $MAINPID
ExecStop=/bin/bin -s TERM $MAINPID
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

Ansible seems to block the process and the server dies, BUT when ran from the command line the exact gunicorn command as in the systemd file it does not hang and works as normal....

Any help or understanding on why this is the case would be greatly appreciated, thanks.

[benoitc]

what do you mean by "hang"? worker timeout or exit immediately? Can you share any log?

[benoitc]

@m57 bump.

[lesourcil]

Hi, i know this thread is very old but i'm experiencing something very similar and i'm trying to run some ansible in python as well and if i run it through gunicorn it hangs after the fast ansible task in the playbook but if run it with uvicorn directly, it works flawlessly. Nothing in the logs, no errors, very difficult to debug when it hangs. We see that the thread is doing something but never finishes. What are your suggestion ?

[jaens]

Ran into a similar issue as well. Anything involving the subprocess module with pipes / capture_output will hang when run inside the Gunicorn master process (in a background thread).

This was with the sync worker so unrelated to #123.