moor
moor copied to clipboard
Persist background tasks in the database / restart background tasks on restart
The tasks in the scheduler are currently not persisted. Problem because:
- Restarts of the
daemonfor upgrades, etc. will kill background tasks without recovery on restart, interrupting the user experience. - LambdaMOO does do this, so we should be doing it for compatibility.
scheduler.rs and task.rs will need some reworking to synchronize their state out to WorldState, which will need new interfaces for this.
And for compatibility, we might need to support the textdump'd background task set from LambdaMOO textdump imports.
This is I think the last task remaining for full MOO 1.8 full compatibility, and other than nailing down stability & bug fixes etc. it's the last thing blocking a 1.0 I think. But it has some challenges
Challenge 1 is the way things are written now, the Scheduler thread doesn't "own" the tasks really in a way that would allow it to look "inside" them to get things like the program counter, VM status, etc. It just uses channels to dispatch instructions to them and get responses, and they run in their own threads. So for it to keep an accounting of the tasks in a way that it can persist information about the task is a bit wonky.
Challenge 2 is performance and logistics related -- when are these tasks sync'd to DB? Too frequently and the DB will be swamped with updates, and will also negatively impact the VM performance. Not frequently enough and we'll lose critical information for restart.
Challenge 3 is determining what exactly to persist. Do we aim to restart all running tasks on restart? Or just, scheduled, explicitly suspended tasks?
Remaining work here:
- A Relbox db backend, instead of just wiredtiger
- Restore tasks mentioned in textdump. And same in reverse.
- Get the task write/commit in a background thread