si icon indicating copy to clipboard operation
si copied to clipboard

feat(rebaser): change quiescent shutdown to reduce missed activity

Open fnichol opened this issue 5 months ago • 0 comments

This change alters the logic that helps a change set "process" task to shut down when no Rebaser requests have been seen over our quiescent_period. Prior to this change there was a shutdown window period where the ChangeSetProcessorTask would not be looking for new Rebaser requests to process while waiting for the SerialDvuTask to end. As a hedge against this scenario the process task handler checks the change set subject just before ending to ensure that if there's at least one request message that we don't ack/delete the task.

In this altered version of a quiescent shutdown we notice the quiet period as before in the Rebaser requests subscription stream. However, now a quiesced_notify tokio::sync::Notify is fired to signal the SerialDvuTask. Then the ChangeSetProcessorTask continues to process any further requests that may show up (remember that after running a "dvu" job, another Rebaser request is often submitted). Meanwhile in the SerialDvuTask, it will continue to run "dvu" jobs as long as the run_dvu_notify has been set (in effect "draining" any pending runs), and only then will check to see if the quiesced_notify has been set. If it has, then it will cancel the quiesced_token which cause SerialDvuTask to return with an Ok(Shutdown::Quiesced) and that same CancellationToken will cause the Naxum app in ChangeSetProcessorTask to be gracefully shut down.

With these changes, the one or two remaining "dvu" jobs will not cause the process task to stop processing further Rebaser requests. For example, let's assuming that the last 2 "dvu" jobs take 8 minutes each. That means that the process task is in a quiescent shutdown for up to the next 8 * 2 = 16 minutes, during which time any further Rebaser requests will also be processed (whereas they may not have been prior to this change).

fnichol avatar Sep 26 '24 17:09 fnichol