Doing cleanup in MirageOS when tender shuts down unikernel

[rand00]

After some discussion with @hannesm it doesn't seem like there is a consistent story of how to do cleanup in a unikernel when it's not an exception that is thrown, but the tender that exits the unikernel.

In my usecase, I was surprised that Mirage_runtime.at_exit didn't run to cleanup my TCP flow, in the case where I Ctrl-c the solo5-spt <unikernel>.

Could Mirage_runtime.at_exit run on any kind of exit?

[reynir]

I looked into the code, and the function registered by at_exit are actually called. https://github.com/mirage/mirage-solo5/blob/main/lib/main.ml#L72-L75

What you're observing could maybe be explained by the call to Lwt.abandon_wakeups - maybe some lwt tasks necessary for actually sending FIN is cancelled and the packet is never sent.

It can also be observed with the following slightly modified mirage-skeleton/tutorial/hello unikernel:

open Lwt.Infix

module Hello (Time : Mirage_time.S) = struct
  let at_exit () =
    Logs.warn (fun f -> f "Oh no, we're exiting!");
    Lwt.return_unit

  let start _time =
    Mirage_runtime.at_exit at_exit;
    let rec loop = function
      | 0 -> Lwt.return_unit
      | n ->
          Logs.info (fun f -> f "hello");
          Time.sleep_ns (Duration.of_sec 1) >>= fun () -> loop (n - 1)
    in
    loop 4
end

$ mirage configure -t && make
...
$ $ solo5-hvt -- dist/hello.hvt 
            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.7.3
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x1dffff)
Solo5:     rodata @ (0x1e0000 - 0x215fff)
Solo5:       data @ (0x216000 - 0x2c0fff)
Solo5:       heap >= 0x2c1000 < stack < 0x20000000
2022-09-14 17:39:32 -00:00: INF [application] hello
2022-09-14 17:39:33 -00:00: INF [application] hello
2022-09-14 17:39:34 -00:00: INF [application] hello
2022-09-14 17:39:35 -00:00: INF [application] hello
2022-09-14 17:39:36 -00:00: WRN [application] Oh no, we're exiting!
Solo5: solo5_exit(0) called

[reynir]

On testing with ctrl-c it does indeed seem to not run the exit handlers on solo5-hvt

$ solo5-hvt -- dist/hello.hvt 
            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.7.3
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x1dffff)
Solo5:     rodata @ (0x1e0000 - 0x215fff)
Solo5:       data @ (0x216000 - 0x2c0fff)
Solo5:       heap >= 0x2c1000 < stack < 0x20000000
2022-09-15 06:05:19 -00:00: INF [application] hello
2022-09-15 06:05:20 -00:00: INF [application] hello
^Csolo5-hvt: Exiting on signal 2

And indeed, the man page for atexit(3) says

Functions registered using atexit() (and on_exit(3)) are not called if a process terminates abnormally because of the delivery of a signal.