retry backoff for meta replication

[everpcpc]

Summary

When some node in cluster restart or in wrong state, log file would be filled with same high-frequency error messages.

[Xuanwo]

I self recommend the crate maintained by myself: https://github.com/Xuanwo/backon

[BohuTANG]

This is in high priority, also cc @lichuang to have a look before @ariesdevil work from fuse engine external location back.

[drmingdrmer]

I self recommend the crate maintained by myself: https://github.com/Xuanwo/backon

Looks great

[Xuanwo]

Since @lichuang & @ariesdevil is busy on other projects, can @ClSlaid take a look on this issue? This issue is marked as prio: high.

[drmingdrmer]

Since @lichuang & @ariesdevil is busy on other projects, can @ClSlaid take a look on this issue? This issue is marked as prio: high.

It'd be nice if @ClSlaid can help on this:

It can be done by wrapping three of the raft-network APIs with some backoff loop. send_append_entries(), send_vote() and send_install_snapshot():

https://github.com/datafuselabs/databend/blob/9e4d7da64f831ab863585c3152af58c905e70041/src/meta/service/src/network.rs#L115