`ERROR | MainnetTransactionProcessor | Critical Exception Processing Transaction` in latest `v22.7.3`

[gamell]

I searched and I've seen several older bugs marked as resolved (1,2) for the same or similar reason, but this just happened to my just updated Besu v22.7.3 + Lighthouse v3.1.0 Rocketpool node, so it doesn't seem fixed.

2022-09-23 23:26:04.216+00:00 | vert.x-worker-thread-18 | ERROR | MainnetTransactionProcessor | Critical Exception Processing Transaction
eth1_1           | org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash 0xe27bc73b3cbcaf5e7f41351838ec84da7be53c4af7f87eacb0f17a04ec59e41e location 0x
eth1_1           | 	at org.hyperledger.besu.ethereum.trie.StoredNode.lambda$load$0(StoredNode.java:133)
eth1_1           | 	at java.base/java.util.Optional.orElseThrow(Optional.java:403)
eth1_1           | 	at org.hyperledger.besu.ethereum.trie.StoredNode.load(StoredNode.java:131)
eth1_1           | 	at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:63)
eth1_1           | 	at org.hyperledger.besu.ethereum.trie.StoredMerklePatriciaTrie.get(StoredMerklePatriciaTrie.java:119)
eth1_1           | 	at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage.getStorageValueBySlotHash(BonsaiWorldStateKeyValueStorage.java:200)
eth1_1           | 	at org.hyperledger.besu.ethereum.bonsai.BonsaiPersistedWorldState.getStorageValueBySlotHash(BonsaiPersistedWorldState.java:381)
eth1_1           | 	at org.hyperledger.besu.ethereum.bonsai.BonsaiLayeredWorldState.getStorageValueBySlotHash(BonsaiLayeredWorldState.java:154)
eth1_1           | 	at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.getStorageValueBySlotHash(BonsaiWorldStateUpdater.java:320)
eth1_1           | 	at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.getStorageValue(BonsaiWorldStateUpdater.java:307)
eth1_1           | 	at org.hyperledger.besu.ethereum.bonsai.BonsaiAccount.getStorageValue(BonsaiAccount.java:208)
eth1_1           | 	at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:213)
eth1_1           | 	at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:213)
eth1_1           | 	at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:213)
eth1_1           | 	at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:213)
eth1_1           | 	at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:213)
eth1_1           | 	at org.hyperledger.besu.evm.operation.SLoadOperation.execute(SLoadOperation.java:61)
eth1_1           | 	at org.hyperledger.besu.evm.EVM.lambda$executeNextOperation$0(EVM.java:87)
eth1_1           | 	at org.hyperledger.besu.evm.tracing.OperationTracer.lambda$static$0(OperationTracer.java:27)
eth1_1           | 	at org.hyperledger.besu.evm.EVM.executeNextOperation(EVM.java:81)
eth1_1           | 	at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:74)
eth1_1           | 	at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:161)
eth1_1           | 	at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:173)
eth1_1           | 	at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:485)
eth1_1           | 	at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:397)
eth1_1           | 	at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:146)
eth1_1           | 	at org.hyperledger.besu.ethereum.transaction.TransactionSimulator.processWithWorldUpdater(TransactionSimulator.java:217)
eth1_1           | 	at org.hyperledger.besu.ethereum.transaction.TransactionSimulator.process(TransactionSimulator.java:160)
eth1_1           | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthCall.resultByBlockHash(EthCall.java:72)
eth1_1           | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterOrBlockHashMethod.latestResult(AbstractBlockParameterOrBlockHashMethod.java:61)
eth1_1           | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractBlockParameterOrBlockHashMethod.handleParamTypes(AbstractBlockParameterOrBlockHashMethod.java:71)
eth1_1           | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthCall.response(EthCall.java:96)
eth1_1           | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
eth1_1           | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:41)
eth1_1           | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
eth1_1           | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:91)
eth1_1           | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$8(JsonRpcExecutorHandler.java:79)
eth1_1           | 	at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
eth1_1           | 	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
eth1_1           | 	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
eth1_1           | 	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
eth1_1           | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
eth1_1           | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
eth1_1           | 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
eth1_1           | 	at java.base/java.lang.Thread.run(Thread.java:833)

I didn't use any special flags for syncing the node, the only current flag I'm using is the new --Xplugin-rocksdb-high-spec-enabled

Thanks!

[daniellehrner]

Does the node progress after you are seeing this error?

[gamell]

Yes, but the same class of error happens around once every hour or so, and I've correlated it to a missed attestation (I can't prove it was the cause, but the timing matches)

[non-fungible-nelson]

Bumping this up with more content.

[xrchz]

I am getting the same error with besu/v22.10.2/linux-x86_64/openjdk-java-19, usually whenever I am submitting transactions on the RPC. The transactions do get submitted, but this error occurs too. MainnetTransactionProcessor | Critical Exception Processing Transaction org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash

[non-fungible-nelson]

Closing for tracking in #4785