Removing leader node

[uber42]

ErrClusterClosed when removing leader, but the node is removed We use backoff on SyncRequestDeleteNode and second attempt fails with ErrClusterNotFound.

It looks like deleting a node is faster than getting a response to a request and exiting the request happens with event Terminated. Reproduced very rarely.

Logs:

msg="[12321:00001] applied REMOVE ccid 0 (79), n00001" component=dragonboat pkg=rsm
msg="[f:1,l:79,t:3,c:79,a:78] [12321:00001] t3 became follower" component=dragonboat pkg=raft
msg="[12321:00001] applied ConfChange Remove for itself" component=dragonboat pkg=dragonboat
msg="leader updated" clid=12321 component=raft leader=0 mds=1 method=LeaderUpdated term=3
msg="raft cluster already closed" clid=12321 component=raft mds=1 <-- first attempt (ErrClusterClosed)
msg="failed to disjoin: failed to delete raft member: cluster not found" component=api <-- out on second attempt (ErrClusterNotFound)

Dragonboat version

v3.3.5