Shutdown all executors API

[artemyarulin]

What do you think about some kind of shutdown all executors API/hook/event/etc?

It's for sure possible to do everything manually and most probably every app would have very customized shutdown behavior, but I guess glommio can simplify things a bit.

Just first random thought that came to mind - have LocalExecutor.request_stop() -> () and LocalExecutor.stop_requested() -> bool and then it would be possible to have shutdown logic in executors smth like that

fn run() {
    loop {
        if Local::stop_requested() {
            break
        }
        doAnotherJob()
    }
}

Or maybe more convenient to have new struct smth like ShutdownHook which could be passed using LocalExecutorBuilder to all executors so that there would be single object to control all the executors smth like

let shutdown = ShutdownHook::new();
for i in 0..5 {
    let builder = LocalExecutorBuilder::new()
        .pin_to_cpu(i)
        .shutdown_hook(shutdown)
        .spawn(doWork);
}
shutdown.request_stop(); // Would set stop_request for all attached executors
// Or maybe even more straight forward
shutdown.stop().await; // Stop and wait until all are done. Would simplify testing

What do you think?

[glommer]

I don't immediately see value in an API that is used to just request a stop. The reason for that, is that the way the executor works is exactly that it executes a future, and when that future finishes the executor stops. So if anything, an object like that could be used to indicate to your main future that it is time to stop, without any executor support.

However, what I do see a lot of value is code to control graceful shutdown: If you Ctrl+C an application, it will essentially just stop all tasks but no special asynchronous code gets invoked to make that happen.

Complex database systems can have equally complex shutdown sequence needs: you may need to flush a cache, sync files, close connections, etc. More than that: some of those things can happen in parallel, but for others there is strict ordering requirements. That is something that I think can fit in a glommio API:

1. execute an asynchronous function on shutdown
2. have that execute a dependency graph of other functions that the application can register.

The implementation of that could likely be an object ApplicationState that implements Future, and that is the Future you pass to the executor (so when ApplicationState exists, so does the executor).

Then there is a method to ApplicationState called, let's say register_shutdown_handler where you can register a ShutdownHandler that has:

1. an async function
2. a named ID.
3. a constraint set.

So for instance, if you have a shutdown handler called "Connections", you may want to execute the cache flush shutdown code After(Connections)

[artemyarulin]

Really like you idea of chained handlers.

Could you please clarify a bit - given simple one executor:

fn run() {
    println!("Started");
    let f = openFile();
    loop {
        appendToFile(f);
    }
    f.close()
    println!("Stopped");
}

How shutdown logic will be integrated in here? I want to close the file at the end as a part of a shutdown. Having totally separate external shutdown function will make it harder to share the state from the executor - in this case file f

[glommer]

In this example you need the file to be kept alive in two places: the shutdown handler, and the normal file operations (since there is no clear way of knowing when to move from one to the other).

So you have to wrap it in an Rc. It would be something like this:

fn run() {
   println!("Started");
   let f = Rc::new(openFile());
   register_shutdown_handler(ShutdownFile::new(f.clone()));
   loop {
       appendToFile(f).await
   }
}