Orderer panics after being restored

[denyeart]

Let's say we have three orderers on a channel, and they have been backed up at different heights: Orderer1 at height 7 Orderer2 at height 8 Orderer3 at height 9

Restore Orderer1 and Orderer2 first.

Orderer2 is leader now: 2022-02-09 13:02:26.183 EST 0118 INFO [orderer.consensus.etcdraft] run -> Raft leader changed: 0 -> 2 channel=mychannel node=1

Orderer1 will get block 8. So far so good.

Then when you bring up Orderer3 there won't be a leader re-election, Orderer3 at height 9 will simply join the existing quorum.

Orderer3 says Orderer2 is still leader. Wait a few minutes to ensure everything is stable and then send a transaction.

Orderer1 and Orderer2 will create block 9.

Orderer3 will complain: 2022-02-09 13:27:54.043 EST 03c6 INFO [orderer.consensus.etcdraft] writeBlock -> Got block [9], expect block [10], this node was forced to catch up channel=mychannel node=3

Send another transaction, then Orderer3 will panic:

2022-02-09 13:28:57.997 EST 0466 PANI [orderer.commmon.multichannel] commitBlock -> [channel: mychannel] Could not append block: unexpected Previous block hash.

I'm posting the logs after the start of Orderer1, Orderer2, and eventually Orderer3 to demonstrate a new leader is not elected at height 9. orderer_logs.zip

[yacovm]

There might be a way around this, but I'm not sure how worthwhile it is to implement it.

We already piggyback the leader's "active nodes" view in order to know whether a config change will result in a quorum loss or not. More specifically, when the leader sends a message to followers, Fabric intercepts that message send event and takes the active node list and makes the message also carry the active nodes list.

Now, at the same time - when a raft follower/candidate receives a heartbeat it sends an ack back to the leader.

Perhaps we could intercept this acknowledgement send of a follower and piggyback the ledger height on top of it. Then, at the leader side, we would call TransferLeadership on it to transfer leadership to the node with the higher ledger height.

Thoughts?

[Param-S]

For the scenario described “the orderer3 joined as follower though its block height greater than other orderers”: It joined as follower as its last log entry term matches with the current quorum/leader’s term. This is the reason for the orderer3 joined as follower.

For the suggestion, “transfer the leadership to the node who’s block height is higher”; we may need to consider the scenario where the node with highest block before joining the quorum and taking over the leadership, what if the previous quorum proceeds and commits few blocks, then there is difference in chain of blocks. In this case, if we take the approach of messages always flowing from current leader to follower then the latest blocks will be overridden.

I think, there is additional logic is required to handle this case to reset the blocks to common level in addition to transfer the leadership based on the block height. We may also need to discuss whether replacing the latest blocks with backup blocks is acceptable or not.