Cannot stop multi-cpu Tornado service on FreeBSD

[rgacote]

I'm unable to stop a Tornado service on FreeBSD when using multi-cpu. When I do service appl start, I see five processes:

• one daemon
• four non-daemon threads

The four non-daemon threads are what I see when I run systemctl start appl on Ubuntu. FreeBSD has the one extra daemon thread.

When I try to stop the FreeBSD service (service app stop) the daemon process and the first (lowest pid) non-daemon process are removed. Three non-daemon processes remain running. Attempting to restart the app says the port is already in use.

I have a shutdown process which logs 'Stopped'. This does not log anything when using three CPUs though it logs properly when using one CPU. The Tornado process that spawns the three CPU worker processes is not killing them.

Again, this works just fine on Ubuntu (although the additional daemon process is missing).

Starting Tornado

My startup application code.

def make_app():
    """Create application instance."""
    args = parse_args(APPNAME, VERSION)
    config = app_config(args)
    if is_pid_running(args):
        raise AlreadyRunningError()
    create_pid(args)
    return tornado.web.Application(
        [
            (r"/payments/v2/", PaymentsHandler),
            (r"/vault/v2/", VaultHandler),
        ],
        **config,
    )


def _main():
    """Create, configure, and start the application."""
    # http://www.tornadoweb.org/en/stable/guide/running.html#running-and-deploying
    global APP, SERVER  # pylint: disable=global-statement

    app = APP = make_app()
    server = SERVER = tornado.httpserver.HTTPServer(
        app,
        xheaders=True,
    )
        server.start(3)

    ioloop = tornado.ioloop.IOLoop.current()

    signal.signal(signal.SIGTERM, exit_handler)
    signal.signal(signal.SIGINT, exit_handler)
    signal.signal(signal.SIGHUP, exit_handler)
    ioloop.start()

async def shutdown():
    """Handle graceful shutdown."""
    if APP:
        settings = APP.settings
        logger = settings["logger"]
        logger.info("Stopping.")
        SERVER.stop()
        # await tornado.gen.sleep(sleep)
        tornado.ioloop.IOLoop.current().stop()
        delete_pid(APP.settings["args"].pid_file)

def exit_handler(sig, frame):
    """Install exit handler."""
    _ = sig
    _ = frame
    tornado.ioloop.IOLoop.instance().add_callback_from_signal(shutdown)

FreeBSD Service file

name=app
RUNUSER="${name}"
RUNGROUP="${name}"

. /etc/rc.subr

rcvar=app_enable
load_rc_config $name

# Set some defaults
app_enable=${app_enable:-"NO"}
app_run_user=${app_run_user:-"$RUNUSER"}
app_run_group=${app_run_group:-"$RUNGROUP"}
app_binary="/usr/local/bin/${name}.pyz"

#pidfile=${app_pidfile:-"/var/run/${name}/${name}.pid"}
pidfile=${app_pidfile:-"/var/run/${name}.pid"}
command="/usr/sbin/daemon"

# -f redirects stdout and stderr to /dev/null
# Don't do that so we can better see start-up errors
# before logs are active.
# TODO: Need to figure out how to restart multi-cpu service.
command_args="-P ${pidfile} -u ${app_run_user} ${app_binary}"
run_rc_command "$1"

Any thoughts?

Versions

• Tornado 6.1
• FreeBSD 12.2
• Python 3.9.4

[bdarnell]

I don't use freebsd so I'm not going to be much help here but I can give you a few pointers.

The 4 non-daemon threads belong to a ThreadPoolExecutor created by the asyncio event loop (and are mainly used for DNS resolution by default). They were changed from daemon threads in Python 3.8 (I think), and they're supposed to be shut down by an atexit hook instead. I've managed to deadlock this with my own atexit hooks, This example doesn't have atexit that shouldn't be an issue. (the signal handlers should be fine), but it does suggest that the new shutdown logic can be fragile sometimes.

I don't know what the daemon thread is; this may be related to something in the python standard library specific to the freebsd platform.

On linux, python signal handlers always run on the main thread. Is that true for freebsd too? If not, you'd have to use add_callback_from_thread instead of add_callback_from_signal.

[Python-37]

I hope this helps, BTW add_callback_from_signal seems not work fine on PyPy on Linux either. Is this code safe to use?@bdarnell

[bdarnell]

I'm not sure about add_callback_from_signal these days. It calls asyncio's call_soon_threadsafe, which was also signal-safe when I looked at it a long time ago, but it's not documented or guaranteed to be signal-safe. We might have to deprecate this method and recommend asyncio's add_signal_handler instead.

[Python-37]

We might have to deprecate this method and recommend asyncio's add_signal_handler instead.

But add_signal_handler is not available on Windows.