[Bug] Leader sending certificate with round far in the future halts the network

[vicsn]

🐛 Bug Report

If a malicious leader sends a certificate with a round set far into the future, the receiving nodes will blindly pass that through try_increment_to_the_next_round(...), repeatedly calling self.storage.increment_to_next_round(...) as the attacker desires.

Steps to Reproduce

We should make sure to have all related malicious leader tests in one branch. For now you can check out this branch and run devnet.sh with a network of size 4.

2023-11-28T14:58:01.379402Z ERROR is_behind_schedule: false, is_peer_far_in_future: true
2023-11-28T14:58:01.379419Z DEBUG Primary is not ready to propose the next round                          
2023-11-28T14:58:01.379439Z  WARN Cannot store a certificate from '127.0.0.1:5001' - Missing certificates for the previous round 57 
in storage (gc = 0)
...
2023-11-28T15:10:36.799805Z  WARN Cannot sign a batch from '127.0.0.1:5001' - Unable to fetch batch certificate - (timeout) deadline
 has elapsed                                                                                                                        
2023-11-28T15:10:36.800164Z DEBUG BFT (timer expired) - Checking for quorum threshold (without the leader)                          

Expected Behavior

The malicious node's certificates should be ignored.

Your Environment

cargo 1.71.1 (7f1d04c00 2023-07-29) macOS