Server controlled retry delay

[gdbelvin]

Replace MinInterval with a server controlled retry delay.

This will allow the server to dynamically and organically rate limit traffic when the server is under load and the queue of mutations is long.

[Zyqsempai]

Sounds interesting, can I help somehow?

[gdbelvin]

Yes, this is important. (And thank you for all of your prior contributions!)

Clients currently use a static duration for controlling how often to enqueue new mutations while they are watching for new updates. If the sequencer gets behind and the queue of mutations grows, clients will pile on more and more mutations exasperating the problem.

The ideal solution would be to have the QueueMutation API return an indicator of how long the queue of mutations currently is and have clients wait that amount of time + jitter.

The sequencer currently reports the depth of the queue behind the UpdateMetrics API. Perhaps the frontend could periodically call this and cache the results?