si
si copied to clipboard
feat(rebaser): change quiescent shutdown to reduce missed activity
This change alters the logic that helps a change set "process" task to shut down when no Rebaser requests have been seen over our quiescent_period
. Prior to this change there was a shutdown window period where the ChangeSetProcessorTask
would not be looking for new Rebaser requests to process while waiting for the SerialDvuTask
to end. As a hedge against this scenario the process task handler checks the change set subject just before ending to ensure that if there's at least one request message that we don't ack/delete the task.
In this altered version of a quiescent shutdown we notice the quiet period as before in the Rebaser requests subscription stream. However, now a quiesced_notify
tokio::sync::Notify
is fired to signal the SerialDvuTask
. Then the ChangeSetProcessorTask
continues to process any further requests that may show up (remember that after running a "dvu" job, another Rebaser request is often submitted). Meanwhile in the SerialDvuTask
, it will continue to run "dvu" jobs as long as the run_dvu_notify
has been set (in effect "draining" any pending runs), and only then will check to see if the quiesced_notify
has been set. If it has, then it will cancel the quiesced_token
which cause SerialDvuTask
to return with an Ok(Shutdown::Quiesced)
and that same CancellationToken
will cause the Naxum app in ChangeSetProcessorTask
to be gracefully shut down.
With these changes, the one or two remaining "dvu" jobs will not cause the process task to stop processing further Rebaser requests. For example, let's assuming that the last 2 "dvu" jobs take 8 minutes each. That means that the process task is in a quiescent shutdown for up to the next 8 * 2 = 16 minutes, during which time any further Rebaser requests will also be processed (whereas they may not have been prior to this change).
