[core] ConsensusAdapter / ConsensusListener is not complete

[gdanezis]

The logic in ConsensusAdapter submit is not complete. Specifically, we submit a transaction after we check that it has not already been sequenced / locks are allocated. The submit logic is here: https://github.com/MystenLabs/sui/blob/2568023e49e3519fcd72ac28f0620c46290ce26b/crates/sui-core/src/consensus_adapter.rs#L148-L190

The issue here is a time of check issue:

• We check that the locks are not set before entering submit.
• Within submit we register an interest in hearing when the transaction is sequenced from consensus.
• Then we may send the transaction through consensus.

However, if we do not send the transaction through consensus, then we may never hear back: what can happen is that it gets sequenced on the validator between the time we enter submit but before we register the listener (done over a channel in another task). As a result the transaction will come, no listener will be registered / activated, and when the listener is registered it will not be activated leading to a spurious timeout.

To resolve this: we should create a central waiting facility for shared object locks being set, rather than for hearing things over the consensus. Since we store locks in a DB if a lock is set, and then a listener registered we can check the lock exists and exit immediately.

[gdanezis]

Node, this issue is less urgent since:

• the client can always retry and this will succeed.
• right now each validator always re-sequences which means that it should receive a sequenced transaction even if one was received before.

[gdanezis]

Also note: it is v possible that the listener and waiter are processed after even our own message is sequenced, and also the listener does not return if locks already exist.