nethermind Fatal error. System.Runtime.InteropServices.SEHException (0x80004005)

Version : 1.14.0-0-60f920b90-20220913/X64-Windows/6.0.8 This node : enode://693dec3469ca7be1fec285e2ada3f5219260ca7f1ce8cf8438fd9264fcbb110acbe7647f2060cc4273c19f95c557a6c179b22fa5f5055cd5d5e6cba4d65479f0@104.217.249.[hide]:30303 RPC modules : Eth, Net, Parity, Personal, Proof, Rpc, Subscribe, Trace, TxPool, Web3 Node address : 0x0f4bac50f3cb88cd263636600f2d1f4fc1ff8844 (do not use as an account) Mem est tx : 82MB Mem est DB : 951MB JSON RPC : http://127.0.0.1:8545 ; http://localhost:8551 Genesis hash : 0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3 External IP : 104.217.249.[hide] Ethereum : tcp://104.217.249.[hide]:30303 Discovery : udp://104.217.249.[hide]:30303 Chainspec : chainspec/foundation.json Chain ID : Mainnet Chain head : 0

2022-09-13 16:14:38.0378|Old Headers 0 / 15379000 | queue 0 | current 0.00bps | total 0.00bps 2022-09-13 16:14:38.8339|Ethash cache miss for block 15379169 (0x23fe2c...ad7131) 2022-09-13 16:14:38.8339|Ethash cache miss for block 15379191 (0xcc4fa9...2d0577) 2022-09-13 16:14:38.8484|Building ethash cache for epoch 512 2022-09-13 16:14:39.0534|Old Headers 0 / 15379000 | queue 0 | current 0.00bps | total 0.00bps 2022-09-13 16:14:40.0533|Old Headers 0 / 15379000 | queue 0 | current 0.00bps | total 0.00bps 2022-09-13 16:14:41.0533|Old Headers 0 / 15379000 | queue 0 | current 0.00bps | total 0.00bps 2022-09-13 16:14:42.0533|Old Headers 0 / 15379000 | queue 0 | current 0.00bps | total 0.00bps 2022-09-13 16:14:43.0532|Old Headers 0 / 15379000 | queue 0 | current 0.00bps | total 0.00bps 2022-09-13 16:14:44.0532|Old Headers 0 / 15379000 | queue 0 | current 0.00bps | total 0.00bps 2022-09-13 16:14:44.6439|Cache for epoch 512 with size 83886016 and seed 31e2acd0944f635daf0b269d5fa2fc565a234c48f27127f9012b3927d073af61 built in 5795ms 2022-09-13 16:14:45.0540|Old Headers 0 / 15379000 | queue 0 | current 0.00bps | total 0.00bps 2022-09-13 16:14:46.0546|Old Headers 960 / 15379000 | queue 5184 | current 0.00bps | total 951.14bps Fatal error. System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception. at Nethermind.Db.Rocks.RocksDbExtensions.GetSpan(RocksDbSharp.RocksDb, Byte[], RocksDbSharp.ColumnFamilyHandle) at Nethermind.Db.DbExtensions.Get[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](Nethermind.Db.IDb, Nethermind.Core.Crypto.Keccak, Nethermind.Serialization.Rlp.IRlpStreamDecoder1<System.__Canon>, Nethermind.Core.Caching.ICache2<Nethermind.Core.Crypto.Keccak,System.__Canon>, Boolean) at Nethermind.Blockchain.BlockTree.FindHeader(Nethermind.Core.Crypto.Keccak, Nethermind.Blockchain.BlockTreeLookupOptions) at Nethermind.Synchronization.ParallelSync.SyncProgressResolver.SearchForFullState(Nethermind.Core.BlockHeader) at Nethermind.Synchronization.ParallelSync.SyncProgressResolver.FindBestFullState() at Nethermind.Synchronization.ParallelSync.MultiSyncModeSelector.TakeSnapshot(Nethermind.Int256.UInt256 ByRef, Int64, Boolean) at Nethermind.Synchronization.ParallelSync.MultiSyncModeSelector.Update() at Nethermind.Synchronization.ParallelSync.MultiSyncModeSelector+<StartAsync>d__50.MoveNext() at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Nethermind.Synchronization.ParallelSync.MultiSyncModeSelector+<StartAsync>d__50, Nethermind.Synchronization, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]].ExecutionContextCallback(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Nethermind.Synchronization.ParallelSync.MultiSyncModeSelector+<StartAsync>d__50, Nethermind.Synchronization, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext(System.Threading.Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Nethermind.Synchronization.ParallelSync.MultiSyncModeSelector+<StartAsync>d__50, Nethermind.Synchronization, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext() at System.Threading.PeriodicTimer+State.Signal(Boolean, System.Threading.CancellationToken) at System.Threading.PeriodicTimer+<>c.<.ctor>b__2_0(System.Object) at System.Threading.TimerQueueTimer.Fire(Boolean) at System.Threading.TimerQueue.FireNextTimers() at System.Threading.UnmanagedThreadPoolWorkItem.ExecuteUnmanagedThreadPoolWorkItem(IntPtr, IntPtr) at System.Threading.UnmanagedThreadPoolWorkItem.ExecuteUnmanagedThreadPoolWorkItem(IntPtr, IntPtr) at System.Threading.UnmanagedThreadPoolWorkItem.System.Threading.IThreadPoolWorkItem.Execute() at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() at System.Threading.Thread.StartCallback()

D:\nethermind-windows-amd64-1.14.0-60f920b> D:\nethermind-windows-amd64-1.14.0-60f920b>

Sep 13 '22 21:09 silmoonsone

Hello @silmooninc!

We reproduced that issue and we are wondering - what CPU did You using on your machine? Suddenly it probably is a case with RocksDB version which we are currently using on nethermind and we are considering upgrade but need to test it properly (RocksDB seems to have some problems with a bit older CPU which does not support AVX2).

Sep 21 '22 10:09 kamilchodola

Hello @silmooninc!

We reproduced that issue and we are wondering - what CPU did You using on your machine? Suddenly it probably is a case with RocksDB version which we are currently using on nethermind and we are considering upgrade but need to test it properly (RocksDB seems to have some problems with a bit older CPU which does not support AVX2).

QQ截图20220921184204

Sep 21 '22 10:09 silmoonsone

Thanks! just checked and this is exactly the same case: Your CPU supports AVX but not AVX2.

We are testing it and still not sure if this works properly yet, so we will let You know about it. Would You be interested checking this version on Your end if we would prepare a beta-version for that?

Sep 21 '22 10:09 kamilchodola

Also - which Windows version are You using?

Sep 21 '22 10:09 kamilchodola

Is it Windows11?

Sep 21 '22 10:09 kamilchodola

Is it Windows11?

Nope, QQ截图20220921185757 Windows Server 2019

Sep 21 '22 10:09 silmoonsone

Is it Windows11?

1.14.x: Windows 11, i9 9900K is okay. but on server it's error. 1.13.x: Same server(server 2019) running normally.

Sep 21 '22 11:09 silmoonsone

So this may be because on one of the 1.13 version we used different RocksDB version and that may be some difference here. In i9 9900K it is supporting AVX2 for now

Sep 21 '22 11:09 kamilchodola

So this may be because on one of the 1.13 version we used different RocksDB version and that may be some difference here. In i9 9900K it is supporting AVX2 for now

I'm a developer of the .NET stack, waiting for your next update, I'm using geth now.

Sep 21 '22 11:09 silmoonsone

Hello, I just had the same issue, I upgraded RocksDB lib from version 6.29.3.26014 to 7.5.3.32564 and problem fixed.

Nethermind version: 1.14.1-11-5b73a2771-20220921/X64-Windows/6.0.9 OS: Windows 10 Pro 21H1 19043.2006 CPU: Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz 3.40 GHz

Some more details here: https://github.com/facebook/rocksdb/issues/8522

Sep 21 '22 21:09 pdoyon

Fixed in #5065

Jan 07 '23 19:01 rubo