sanic
sanic copied to clipboard
Shutdown never finishes when background tasks are cancelled but need some time to finish
Is there an existing issue for this?
- [X] I have searched the existing issues
Describe the bug
I have an app that has background task, and this background task needs some time to finish once shutdown (initiated by SIGINT
) is requested. Unfortunately, this seldom works as expected - often it never finishes and gets stuck in shutdown_tasks()
.
The are couple problems that I recognize but could not find a solution yet. The part of the code that is handling shutdown:
https://github.com/sanic-org/sanic/blob/acb29c9dc4d6ba3a453a18e30f0664ba6772a9b4/sanic/app.py#L1910-L1918
- by default
increment = 0.1
, eventually it becomes negative due to imprecise nature of floats, thus loop will never finish as long as there are running tasks. Actually there is no "safe" value for anincrement
as long as eithertimeout
orincrement
are not integers. - there is always running (background) task as asyncio loop is killed at this point and the while loop is in eternal cycle (and
get_running_loop()
produces an exception), so it eats 100% CPU as well.
Since background_task()
is perfectly terminates by SIGINT
when run directly using asyncio.run(background_task())
(after cancellation), I believe the problem is in Sanic
.
The following snippet could be used to reproduce the problem (I ran it on Debian 12, Python 3.11):
Code snippet
import asyncio
import signal
from sanic import Sanic
app = Sanic("sanic-bug")
app.config.GRACEFUL_SHUTDOWN_TIMEOUT = 5
bg_task: asyncio.Task | None = None
async def background_task():
print("Background task running")
try:
print("Sleep one")
await asyncio.sleep(30)
except asyncio.CancelledError:
# This get executed when Ctrl-C is hit as task is cancelled
print("Sleep one cancelled")
try:
print(f"Sleep two {asyncio.get_running_loop()}")
# This never finishes as long as timeout is > 0
# and Sanic.shutdown_tasks() is in eternal loop,
await asyncio.sleep(5)
except asyncio.CancelledError:
print("Sleep two cancelled")
asyncio.current_task().uncancel()
except BaseException as e:
print(f"Oops: {e!r}")
print("Background task finished")
return
@app.before_server_start
async def prepare_start(*args):
print("Starting background task")
_ = app.add_task(background_task(), name="background-task", register=True)
print("Started background task")
if __name__ == "__main__":
app.run(host="0.0.0.0", single_process=True)
The output:
Starting background task
Started background task
Background task running
Sleep one
[2024-01-19 13:48:59 +0100] [857208] [INFO] Starting worker [857208]
^CSleep one cancelled
Sleep two <uvloop.Loop running=True closed=False debug=False>
[2024-01-19 13:49:02 +0100] [857208] [INFO] Stopping worker [857208]
At this point it hangs forever. This also affect multi-process mode.
Expected Behavior
I would expect that asyncio loop is not touched until and unless all background tasks are finished or when GRACEFUL_SHUTDOWN_TIMEOUT
expires, in the latter case it could be forcibly killed.
How do you run Sanic?
As a script (app.run
or Sanic.serve
)
Operating System
Linux
Sanic Version
23.12.1
Additional context
No response