Invalid Blocks after node restart (crash?) v4.0.0

[Stonygan]

Summary

Tonight 2 independent nodes crashed on the same block (#3470797) and after the restart both reported the valid block #3470790 (hash: 71d037fbaa1e7d04254dfc49fb2816f91766c50b639d47f0f60c451397f76c0b) as invalid and jumped back to #3470789. The nodes then remained stuck on this block until one of the two nodes mined a new block with its own chain. #3470790 with hash e477d9c4ca899408fc8073f686c7e9c9aa8e7444a3a9e381947ab2c7ab1f9c3f, which the other node then also jumped to. The node recognizes that there is a longer chain, but does not jump to it even several hours later. A reconsiderblock of the invalid block does not produce an error and no output in the defi-cli, but is still displayed as invalid in the debug.log. An invalidateblock of the incorrectly created block marks it as invalid, a reconsiderblock on the valid block that was executed before the invalidate without error output then reports:

~/.defi/defi-cli reconsiderblock 71d037fbaa1e7d04254dfc49fb2816f91766c50b639d47f0f60c451397f76c0b
error code: -20
error message:
 (code 16)

The node can no longer switch to the correct chain. We have the same behavior with a normal restart of the node with defi-cli stop. This is also the reason why the snapshots (all those created after the hard fork) no longer work, as they always recognize the following block as invalid.

Maybe others can also observe this behavior, these are the first findings that we noticed in the first few days after the hardfork on the mainnet. Anyone who can contribute further details, please feel free to add them.

Steps to Reproduce

Stop the node and start again after few minutes.

Environment

[Please fill all of the following or NA if not applicable]

• Node Version: v4.0.0 and v4.0.1
• Block height on bug if applicable: see above
• TX or TX type on bug if applicable: see above
• OS with version: Debian 11/12/ubuntu 22
• Any other relevant environment info:

Notecrash in the debug.log:

2023-11-19T04:41:34Z UpdateTip: new best=9fb3beb9ed5f4fcba08097b241f3bc6a6eb9d28cd42260e98c02f4c89b2b77b0 height=3470797 version=0x20000000 log2_work=87.942014 tx=27192192 date='2023-11-19T04
:41:47Z' progress=1.000000 cache=0.0MiB(39txo)
...............................................................................................................................................................................................
...............................................................................................................................................................................................
...............................................................................................................................................................................................
...............................................................................................................................................................................................
.........