`TrieException` (node is missing) error on Gnosis Chain archive node

[varasev]

Hi,

We have a problem with an archive node for Blockscout on Gnosis Chain. Working on v1.12.3. It caught the TrieException error twice during a few months. Restart didn't help. Resyncing from scratch takes long time and network traffic, so we had to temporarily switch our Nethermind archive node to OpenEthereum.

Error log:

ethereum    | 2022-02-28 23:33:03.1438|BlockchainProcessor encountered an exception. Nethermind.Trie.TrieException: Node 0x5b3773ce8d5f355ac3c0a77718152beb777120e103864ac09c3927a82763c1c1 is missing from the DB
ethereum    |    at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak, IKeyValueStore keyValueStore) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 331
ethereum    |    at Nethermind.Trie.Pruning.ReadOnlyTrieStore.LoadRlp(Keccak hash) in /src/Nethermind/Nethermind.Trie/Pruning/ReadOnlyTrieStore.cs:line 41
ethereum    |    at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree) in /src/Nethermind/Nethermind.Trie/TrieNode.cs:line 234
ethereum    |    at Nethermind.Trie.PatriciaTree.Run(Span`1 updatePath, Int32 nibblesCount, Byte[] updateValue, Boolean isUpdate, Boolean ignoreMissingDelete, Keccak startRootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 401
ethereum    |    at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 319
ethereum    |    at Nethermind.State.StateTree.Get(Address address, Keccak rootHash) in /src/Nethermind/Nethermind.State/StateTree.cs:line 49
ethereum    |    at Nethermind.State.StateProvider.GetState(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 659
ethereum    |    at Nethermind.State.StateProvider.GetAndAddToCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 674
ethereum    |    at Nethermind.State.StateProvider.GetThroughCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 695
ethereum    |    at Nethermind.State.StateProvider.GetCodeHash(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 311
ethereum    |    at Nethermind.State.IReadOnlyStateProvider.IsContract(Address address) in /src/Nethermind/Nethermind.State/IReadOnlyStateProvider.cs:line 39
ethereum    |    at Nethermind.Evm.TransactionProcessing.ReadOnlyTransactionProcessor.IsContractDeployed(Address address) in /src/Nethermind/Nethermind.Evm/TransactionProcessing/ReadOnlyTransactionProcessor.cs:line 55
ethereum    |    at Nethermind.Blockchain.Contracts.Contract.ConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 111
ethereum    |    at Nethermind.Consensus.AuRa.Contracts.TransactionPermissionContract.PermissionConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor) in /src/Nethermind/Nethermind.Consensus.AuRa/Contracts/TransactionPermissionContract.cs:line 135
ethereum    |    at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call(CallInfo callInfo) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 104
ethereum    |    at Nethermind.Blockchain.Contracts.Contract.IConstantContract.Call[T](CallInfo callInfo) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 43
ethereum    |    at Nethermind.Blockchain.Contracts.Contract.IConstantContract.Call[T](BlockHeader parentHeader, String functionName, Address sender, Object[] arguments) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 52
ethereum    |    at Nethermind.Consensus.AuRa.Contracts.TransactionPermissionContract.ContractVersion(BlockHeader blockHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Contracts/TransactionPermissionContract.cs:line 82
ethereum    |    at Nethermind.Consensus.AuRa.Contracts.VersionedContract`1.ResolveVersion(BlockHeader blockHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Contracts/VersionedContract.cs:line 57
ethereum    |    at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetVersionedContract(BlockHeader blockHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Transactions/TxPermissionFilter.cs:line 109
ethereum    |    at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetPermissionsFromContract(Transaction tx, BlockHeader parentHeader, ValueTuple`2& key) in /src/Nethermind/Nethermind.Consensus.AuRa/Transactions/TxPermissionFilter.cs:line 79
ethereum    |    at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetPermissions(Transaction tx, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Transactions/TxPermissionFilter.cs:line 65
ethereum    |    at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.IsAllowed(Transaction tx, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Transactions/TxPermissionFilter.cs:line 55
ethereum    |    at Nethermind.Consensus.Transactions.CompositeTxFilter.IsAllowed(Transaction tx, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus/Transactions/CompositeTxFilter.cs:line 38
ethereum    |    at Nethermind.Consensus.AuRa.AuRaBlockProcessor.CheckTxPosdaoRules(AddingTxEventArgs args) in /src/Nethermind/Nethermind.Consensus.AuRa/AuRaBlockProcessor.cs:line 161
ethereum    |    at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ValidateTxs(Block block) in /src/Nethermind/Nethermind.Consensus.AuRa/AuRaBlockProcessor.cs:line 126
ethereum    |    at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ValidateAuRa(Block block) in /src/Nethermind/Nethermind.Consensus.AuRa/AuRaBlockProcessor.cs:line 103
ethereum    |    at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options) in /src/Nethermind/Nethermind.Consensus.AuRa/AuRaBlockProcessor.cs:line 89
ethereum    |    at Nethermind.Blockchain.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Blockchain/Processing/BlockProcessor.cs:line 224
ethereum    |    at Nethermind.Blockchain.Processing.BlockProcessor.Process(Keccak newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Blockchain/Processing/BlockProcessor.cs:line 119
ethereum    |    at Nethermind.Blockchain.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch processingBranch, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Blockchain/Processing/BlockchainProcessor.cs:line 383
ethereum    |    at Nethermind.Blockchain.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Blockchain/Processing/BlockchainProcessor.cs:line 295
ethereum    |    at Nethermind.Blockchain.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Blockchain/Processing/BlockchainProcessor.cs:line 238
ethereum    |    at System.Threading.Tasks.Task.InnerInvoke()
ethereum    |    at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
ethereum    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
ethereum    | --- End of stack trace from previous location ---
ethereum    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
ethereum    |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)

[LukaszRozmej]

You can run debug_resetHead to a block you definitely have state for and sync from this block.

[raymondjacobson]

This issue prevents the node from coming online, so the RPC isn't live to send a debug_resetHead. Is there another way to send it via config / startup?

[LukaszRozmej]

@raymondjacobson sorry for late reply, was this resolved with a resync? Otherwise we can investigate - the order was changed for the merge as a workaround of some issues there, we can think if we can make it more sophisticated.