falcon icon indicating copy to clipboard operation
falcon copied to clipboard

Graceful shutdown on kubernetes

Open mattbooks opened this issue 3 weeks ago • 1 comments

After reading #71, testing and reading the code, my understanding is that falcon expects a SIGINT to trigger a graceful shutdown. However, in kubernetes, SIGTERM is sent to trigger shutdown. When this happens, falcon immediately shuts down children, exits with code 1 and logs several errors. I'm not totally sure what the impact is on code running in the children.

Is it possible to enable a clean shutdown with SIGTERM?

SIGINT behaviour:

❯ bundle exec falcon host
  0.0s     info: Falcon::Command::Host [oid=0x9d0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:45 -0800]
               | Falcon Host v0.52.4 taking flight!
               | - Configuration: falcon.rb
               | - To terminate: Ctrl-C or kill 92915
               | - To reload: kill -HUP 92915
 0.01s     info: Async::Container::Notify::Console [oid=0x9e0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:45 -0800]
               | {status: "Initializing controller..."}
 0.01s     info: Falcon::Service::Server [oid=0x9f0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:45 -0800]
               | Preloading preload.rb...
  1.6s     info: Falcon::Service::Server [oid=0x9f0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Starting my-app on #<Async::HTTP::Endpoint http://0.0.0.0:3000/ {protocol: Async::HTTP::Protocol::HTTP11}>
  1.6s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Controller starting...
  1.6s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Starting container...
 1.64s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Waiting for startup...
 1.65s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Finished startup.
 1.65s     info: Async::Container::Notify::Console [oid=0x9e0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | {ready: true, size: 11}
 1.65s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Controller started...
34.39s     info: Async::Container::Forked [oid=0x2ea8] [ec=0x9d8] [pid=92915] [2025-12-01 10:09:20 -0800]
               | Stopping container...
               | {
               |   "timeout": true,
               |   "caller": [...]
               | }
34.39s     info: Async::Container::Group [oid=0x2eb0] [ec=0x9d8] [pid=92915] [2025-12-01 10:09:20 -0800]
               | Sending interrupt to 11 running processes...
34.53s     info: Async::Container::Forked [oid=0x2ea8] [ec=0x9d8] [pid=92915] [2025-12-01 10:09:20 -0800]
               | Group has stopped.

SIGTERM behaviour:

❯ bundle exec falcon host
  0.0s     info: Falcon::Command::Host [oid=0x9d0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:22 -0800]
               | Falcon Host v0.52.4 taking flight!
               | - Configuration: falcon.rb
               | - To terminate: Ctrl-C or kill 99553
               | - To reload: kill -HUP 99553
 0.01s     info: Async::Container::Notify::Console [oid=0x9e0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:22 -0800]
               | {status: "Initializing controller..."}
 0.01s     info: Falcon::Service::Server [oid=0x9f0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:22 -0800]
               | Preloading preload.rb...
 2.08s     info: Falcon::Service::Server [oid=0x9f0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Starting my-app on #<Async::HTTP::Endpoint http://0.0.0.0:3000/ {protocol: Async::HTTP::Protocol::HTTP11}>
 2.08s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Controller starting...
 2.08s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Starting container...
 2.09s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Waiting for startup...
 2.09s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Finished startup.
 2.09s     info: Async::Container::Notify::Console [oid=0x9e0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | {ready: true, size: 2}
 2.09s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Controller started...
10.37s     info: Async::Container::Forked [oid=0x2e18] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:33 -0800]
               | Stopping container...
               | {
               |   "timeout": false,
               |   "caller": [...]
               | }
10.37s     info: Async::Container::Group [oid=0x2e20] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:33 -0800]
               | Sending terminate to 2 running processes...
10.37s    error: Async::Container::Forked::Child [ec=0x2df8] [pid=99556] [2025-12-01 10:44:33 -0800]
               | SIGTERM
               |   Async::Container::Terminate: SIGTERM
               |   → /Users/matt.books/.local/share/mise/installs/ruby/3.4.2/lib/ruby/gems/3.4.0/gems/async-2.34.0/lib/async/scheduler.rb:525 in 'Thread.handle_interrupt' [...]
10.37s    error: Async::Container::Forked::Child [ec=0x2e08] [pid=99557] [2025-12-01 10:44:33 -0800]
               | SIGTERM
               |   Async::Container::Terminate: SIGTERM
               |   → /Users/matt.books/.local/share/mise/installs/ruby/3.4.2/lib/ruby/gems/3.4.0/gems/async-2.34.0/lib/async/scheduler.rb:525 in 'Thread.handle_interrupt' [...]
10.38s    error: Async::Container::Forked [oid=0x2e18] [ec=0x2e28] [pid=99553] [2025-12-01 10:44:33 -0800]
               | Child exited with error!
               | {
               |   "status": "pid 99557 exit 1",
               |   "running": false
               | }
10.38s    error: Async::Container::Forked [oid=0x2e18] [ec=0x2e30] [pid=99553] [2025-12-01 10:44:33 -0800]
               | Child exited with error!
               | {
               |   "status": "pid 99556 exit 1",
               |   "running": false
               | }
10.38s     info: Async::Container::Forked [oid=0x2e18] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:33 -0800]
               | Group has stopped.

mattbooks avatar Dec 01 '25 19:12 mattbooks

Thanks for your detailed report. I need to think about the best approach.

ioquatix avatar Dec 06 '25 08:12 ioquatix