Chain Stopped Producing Blocks when Validator was Removed

[yohanelly95]

System information

Geth version: v1.10.3 Quorum version: v24.4.1 OS & Version: Linux

Expected behaviour

I used a simple contract that adds and removes validators from a list. The quorum chain uses this contract to getValidators().

I added a new validator to the chain successfully ie, now the chain has 6 Validators (of a minimum requirement of 4). When I removed the recently added validator from the contract the chain should continue to produce blocks as long as the minimum number of validators are running and proposing blocks.

Actual behaviour

After removing the validator, ie, 5 active validators on the chain. The chain stopped producing blocks completely and cannot recover even though I have 5 validators running.

Attaching logs of errors and warnings that are shown in the node logs.

ERROR[08-30|07:54:37.042] QBFT: invalid PREPARE message digest     address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared      msg.code=19 msg.source=0x6F1013A68bcfc9dA74FB83408CA7Db32e7731D75 msg.round=4 msg.sequence=11484

INFO [08-30|07:54:37.036] QBFT: Correctly decoded SignedRoundChangePayload p="RoundChange {seq=11484, round=4, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
INFO [08-30|07:54:37.036] QBFT: Correctly decoded SignedRoundChangePayload p="RoundChange {seq=11484, round=4, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
INFO [08-30|07:54:37.036] QBFT: Correctly decoded SignedRoundChangePayload p="RoundChange {seq=11484, round=4, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
INFO [08-30|07:54:37.036] QBFT: Correctly decoded SignedRoundChangePayload p="RoundChange {seq=11484, round=4, pr=<nil>, pv=0x0000000000000000000000000000000000000000000000000000000000000000}"
ERROR[08-30|07:54:37.036] QBFT: Error List() Signed Payload        err="rlp: end of list"
TRACE[08-30|07:54:37.037] QBFT: confirmation Formula used ceil(2N/3) address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared
INFO [08-30|07:54:37.037] QBFT: handle PREPARE message             address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared      msg.code=19 msg.source=0xc16E2a9092e92210cE55F05023D1c45d01689037 msg.round=4 msg.sequen
ce=11484 prepares.count=0 quorum=4
ERROR[08-30|07:54:37.037] QBFT: invalid PREPARE message digest     address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared      msg.code=19 msg.source=0xc16E2a9092e92210cE55F05023D1c45d01689037 msg.round=4 msg.sequen
ce=11484
DEBUG[08-30|07:54:37.038] consensus message was handled by consensus engine id=9c30927cd6e5da5a conn=inbound            msg=19 quorumConsensusProtocolName=istanbul err=nil
DEBUG[08-30|07:54:37.039] consensus message was handled by consensus engine id=04d9d8ed628abf20 conn=inbound            msg=19 quorumConsensusProtocolName=istanbul err=nil
DEBUG[08-30|07:54:37.039] consensus message was handled by consensus engine id=19bb06c402bc4098 conn=inbound            msg=19 quorumConsensusProtocolName=istanbul err=nil
DEBUG[08-30|07:54:37.039] consensus message was handled by consensus engine id=19bb06c402bc4098 conn=inbound            msg=19 quorumConsensusProtocolName=istanbul err=nil
TRACE[08-30|07:54:37.040] QBFT: confirmation Formula used ceil(2N/3) address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared
INFO [08-30|07:54:37.040] QBFT: handle PREPARE message             address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared      msg.code=19 msg.source=0xFDd781c436c2FB360F7458AF64ACEAF5Ee13e6e5 msg.round=4 msg.sequen
ce=11484 prepares.count=0 quorum=4
TRACE[08-30|07:54:37.040] QBFT: confirmation Formula used ceil(2N/3) address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared
TRACE[08-30|07:54:37.040] QBFT: confirmation Formula used ceil(2N/3) address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared
DEBUG[08-30|07:54:37.040] QBFT: accepted PREPARE messages          address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared      msg.code=19 msg.source=0xFDd781c436c2FB360F7458AF64ACEAF5Ee13e6e5 msg.round=4 msg.sequen
ce=11484 prepares.count=1 quorum=4
DEBUG[08-30|07:54:37.040] consensus message was handled by consensus engine id=9c30927cd6e5da5a conn=inbound            msg=19 quorumConsensusProtocolName=istanbul err=nil
TRACE[08-30|07:54:37.041] QBFT: confirmation Formula used ceil(2N/3) address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared
INFO [08-30|07:54:37.041] QBFT: handle PREPARE message             address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared      msg.code=19 msg.source=0xDa06ecA6f65f05b573820BF781819dC4D41f6a3d msg.round=4 msg.sequen
ce=11484 prepares.count=1 quorum=4
ERROR[08-30|07:54:37.041] QBFT: invalid PREPARE message digest     address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared      msg.code=19 msg.source=0xDa06ecA6f65f05b573820BF781819dC4D41f6a3d msg.round=4 msg.sequen
ce=11484
DEBUG[08-30|07:54:37.041] consensus message was handled by consensus engine id=19bb06c402bc4098 conn=inbound            msg=19 quorumConsensusProtocolName=istanbul err=nil
TRACE[08-30|07:54:37.041] QBFT: confirmation Formula used ceil(2N/3) address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared
INFO [08-30|07:54:37.041] QBFT: handle PREPARE message             address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared      msg.code=19 msg.source=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B msg.round=4 msg.sequen
ce=11484 prepares.count=1 quorum=4
TRACE[08-30|07:54:37.041] QBFT: confirmation Formula used ceil(2N/3) address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared
TRACE[08-30|07:54:37.041] QBFT: confirmation Formula used ceil(2N/3) address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared
DEBUG[08-30|07:54:37.041] QBFT: accepted PREPARE messages          address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared      msg.code=19 msg.source=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B msg.round=4 msg.sequen
ce=11484 prepares.count=2 quorum=4
TRACE[08-30|07:54:37.042] QBFT: confirmation Formula used ceil(2N/3) address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared
INFO [08-30|07:54:37.042] QBFT: handle PREPARE message             address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared      msg.code=19 msg.source=0x6F1013A68bcfc9dA74FB83408CA7Db32e7731D75 msg.round=4 msg.sequen
ce=11484 prepares.count=2 quorum=4
ERROR[08-30|07:54:37.042] QBFT: invalid PREPARE message digest     address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=4 current.sequence=11484 state=Preprepared      msg.code=19 msg.source=0x6F1013A68bcfc9dA74FB83408CA7Db32e7731D75 msg.round=4 msg.sequen
ce=11484

and finally

WARN [08-30|08:02:37.046] QBFT: ignore PRE-PREPARE message from non proposer address=0x0D638cdc26D8AE3325bF4EBa49992e28b0f4Af9B current.round=6 current.sequence=11484 state="Accept request" msg.code=18 msg.source=0xDa06ecA6f65f05b573820BF781819dC4D41f6a3d msg.round=6 msg.sequence=11484 proposal.number=11484 proposal.hash=0x836e3924a7dc9c97af4c523055ff8bb6d52331fff0bc702232f92dce15ce00b2 proposer=0x8522537600244d9d45C39947191a1Eec1fB19A70

The validator that was removed was validator 0x8522537600244d9d45C39947191a1Eec1fB19A70 which was the one that was to propose the current block/proposed last block. I am not sure how to handle this when relying on smart contracts for validator selection.

Steps to reproduce the behaviour

Use a smart contract to add/remove validators. Add a new validator and after it produces some blocks remove the validator.