Task effect
This is a proposal for a new effect type to allow the Ra leader to run and monitor "tasks" on behalf of the state machine. Tasks are run in their own processes that are spawned using spawn_link to ensure that we can report back to the state machine when a task terminates abnormally as well as terminate running tasks when the leader exits.
the type of the effect could be {task, TaskId :: term(), {module(), Function :: atom(), Args :: list(), Options :: list()}
The TaskId can be any term that uniquely identifies the task to the state machine. Most likely it will be a tuple of some sorts derived from data held by the state machine and possibly the command index and/or term.
The options list will initially be empty but could include things like a retry configuration or a task delay.
Ra will capture the return value of the MFA and append the following task_result command to the Ra log including the return value with the success indicator. If the function throws any exception the task_result will be set to failure and the error will be included instead of the return value.
{task_result, TaskId :: term(), success | failure, ReturnValueOrErrorReason :: term()}
If a state machine wants to terminate a pending task for any reason it can emit the {terminate_task, TaskId::term()} effect. A success/failure command may still be reported back to the state machine due to timing issues.
When a leader is made to step down it will terminate all pending tasks. The state machine should re-issue any pending tasks on state_enter(leader, just like needs to be done for monitors and timers. This means that tasks may need to be idempotent. Until a successful task_result has been returned tasks may or may not have been executed.