moor icon indicating copy to clipboard operation
moor copied to clipboard

Persist background tasks in the database / restart background tasks on restart

Open rdaum opened this issue 2 years ago • 2 comments

The tasks in the scheduler are currently not persisted. Problem because:

  • Restarts of the daemon for upgrades, etc. will kill background tasks without recovery on restart, interrupting the user experience.
  • LambdaMOO does do this, so we should be doing it for compatibility.

scheduler.rs and task.rs will need some reworking to synchronize their state out to WorldState, which will need new interfaces for this.

And for compatibility, we might need to support the textdump'd background task set from LambdaMOO textdump imports.

rdaum avatar Sep 23 '23 15:09 rdaum

This is I think the last task remaining for full MOO 1.8 full compatibility, and other than nailing down stability & bug fixes etc. it's the last thing blocking a 1.0 I think. But it has some challenges

Challenge 1 is the way things are written now, the Scheduler thread doesn't "own" the tasks really in a way that would allow it to look "inside" them to get things like the program counter, VM status, etc. It just uses channels to dispatch instructions to them and get responses, and they run in their own threads. So for it to keep an accounting of the tasks in a way that it can persist information about the task is a bit wonky.

Challenge 2 is performance and logistics related -- when are these tasks sync'd to DB? Too frequently and the DB will be swamped with updates, and will also negatively impact the VM performance. Not frequently enough and we'll lose critical information for restart.

Challenge 3 is determining what exactly to persist. Do we aim to restart all running tasks on restart? Or just, scheduled, explicitly suspended tasks?

rdaum avatar Jan 30 '24 13:01 rdaum

Remaining work here:

  • A Relbox db backend, instead of just wiredtiger
  • Restore tasks mentioned in textdump. And same in reverse.
  • Get the task write/commit in a background thread

rdaum avatar Jul 02 '24 22:07 rdaum