dymint
dymint copied to clipboard
nodes can start with wrong last submitted height from the Hub after rotation
when restarting, nodes may miss state updates. it may happen the state update was sent before the node was restarted but accepted after the node starts, and therefore the node starts with an old last submitted height. this was an issue for sequencers, because they were not listening for state updates and getting a wrong batch height when it was restarted waiting for submission confirmation from the hub. we added a modification for the sequencers to also subscribe to state update events, which updates the last submitted height in case the previous issue happened. however, when there is a rotation, there is an edge case where the full-node that will start as a new sequencer was not synced. in this case, if it starts with an old submitted height from the hub, because at the moment of restart the last state update was not accepted yet, it will start producing blocks that were actually already submitted by the old sequencer and this will cause a non-recoverable issue, because even if it receives the new state update after, it will already have produced blocks that it shouldnt. this issue is related to #1092.