Faling engine-api hive test: Re-org to Previously Validated Sidechain Payload

[fgimenez]

Only with --experimental.engine

This is hive log:

>> (cf7563b7) {"jsonrpc":"2.0","id":51,"method":"engine_newPayloadV1","params":[{"parentHash":"0x531eecf1192a7b1b2df16faf91fadfd2859098d3515138942d9633eff525b926","feeRecipient":"0x0000000000000000000000000000000000000000","stateRoot":"0x2f335548d7a5ae42ffcd96dd958a7a26796c2039d78cfb837e62d1a2f8e286d5","receiptsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","prevRandao":"0x85f546d4904f81b7e30d4d1edba008cdce5407ecf66a1b9995fbcf1908f3c419","blockNumber":"0xb","gasLimit":"0x2fefd8","gasUsed":"0x0","timestamp":"0x123f","extraData":"0x726574682f76312e302e362f6c696e7578","baseFeePerGas":"0xdda1b8c","blockHash":"0x4f9df799087267b075bff69416e3359ed9f92262b6ba4b5e3c9db45031027636","transactions":[],"withdrawals":null}]}            
<< (cf7563b7) {"jsonrpc":"2.0","id":51,"result":{"status":"VALID","latestValidHash":"0x4f9df799087267b075bff69416e3359ed9f92262b6ba4b5e3c9db45031027636","validationError":null}}      
>> (cf7563b7) {"jsonrpc":"2.0","id":52,"method":"engine_forkchoiceUpdatedV1","params":[{"headBlockHash":"0x4f9df799087267b075bff69416e3359ed9f92262b6ba4b5e3c9db45031027636","safeBlockHash":"0x0000000000000000000000000000000000000000000000000000000000000000","finalizedBlockHash":"0x0000000000000000000000000000000000000000000000000000000000000000"},null]}              
<< (cf7563b7) {"jsonrpc":"2.0","id":52,"result":{"payloadStatus":{"status":"VALID","latestValidHash":"0x4f9df799087267b075bff69416e3359ed9f92262b6ba4b5e3c9db45031027636","validationError":null},"payloadId":null}}                                                                                                                                                             
>> (cf7563b7) {"jsonrpc":"2.0","id":35,"method":"eth_getBlockByNumber","params":["0xb",false]}                                                                                           
<< (cf7563b7) {"jsonrpc":"2.0","id":35,"error":{"code":-32001,"message":"block not found: number 0xb"}}             

No noticeable errors on the node logs. The results are flaky, it passes around 16.66%.

The flakiness seems related to the persistence task, extending the in-memory buffer size to 6 fixes it.

Attached logs for hive and the node in success and failure.

failed-client.log successful-client.log successful-hive.log failed-hive.log