Integration testing fails with catastrophic ISS

[imalygin]

Description

This issue caused by the #14356 .

The ISS detection tool is working with a SignedState instance, which, in its turn, contains an instance of MerkleRoot. The commit changes the root. It may very well be that the tool is working with the old root, which is why it has a different hash. This is supported by several pieces of evidence:

• The hash mnemonics of the root (cloud-hundred-soup-mistake) can't be found anywhere in the logs except in the ISS detection tool report.
• The logs contain multiple messages like this one:

2024-08-01 11:19:29.773 189      ERROR EXCEPTION        <platformForkJoinThread-1> DefaultSignedStateSentinel: Old signed state detected. The most likely causes are either that the node has gotten stuck or that there has been a memory leak.

• We see ISS despite the fact that all nodes report the same root hash for rounds 205825 and 205828.

The only doubt I have about this theory is this: why didn't it happen on all of the nodes? Yes, it calculates a wrong hash, but it does that consistently. The only possible explanation I have is that in some cases, we calculate a root hash for the correct root. That is, if the node could finish the migration on time, it would have a correct root. Otherwise, the root would be incorrect.

Steps to reproduce

Run a test that involves a restart

Additional context

No response

Hedera network

previewnet

Version

v0.53

Operating system

None