Figure out why the extra functional tests are failing

[SamHSmith]

Here is the conversation we had already on telegram.

Roman Shanin, [08/08/2024 15:16]
what exact problem you are talking about?

mingler, [08/08/2024 15:17]
try running unstable_network_8_peers_1_fault. Once you start seeing Transaction already in blockchain test fails

mingler, [08/08/2024 15:18]
happens every time I run it

mingler, [08/08/2024 15:19]
the same happens with unstable_network_5_peers_1_fault

Sam H. Smith, [08/08/2024 15:19]
Aah

Sam H. Smith, [08/08/2024 15:20]
I think this might have to do with the transaction gossip issue that roman recently opened.

mingler, [08/08/2024 15:20]
yeah, looks like gossiping issue

Sam H. Smith, [08/08/2024 15:25]
This is my theory. We check if transactions are in the blockchain when they are put in the queue. And potentially when they come out of the queue, but less sure on that one. This is a very slow operation, which is why we have the sumeragi tx cache. However that means that a transactions can be in the transaction cache and not be checked against being in the blockchain already.

Sam H. Smith, [08/08/2024 15:26]
When the leader creates a block I think we prune transactions that are already in the chain.

Roman Shanin, [08/08/2024 15:28]
> And potentially when they come out of the queue, but less sure on that one.
Yes, we check that transaction is not in the blockchain on pop from the queue

Roman Shanin, [08/08/2024 15:30]
It could be that gossips are piling up in communication channel (outgoing peer message queue, socket, incoming peer message queue) l and by the time gossip is received it's already in the chain.

Roman Shanin, [08/08/2024 15:34]
At least from my experiments with dynamic commit time i've noticed like 20 seconds delay between sending block created message and receiving it by the other peer.

Sam H. Smith, [08/08/2024 15:35]
Oh wow

Sam H. Smith, [08/08/2024 15:35]
That's bad