Externalities not allowed to fail within runtime

[Unrender]

Good morning,

This isn't the first time I've had this error, but I don't know where it could come from. And it occurs at different times. The first time, the node had been running for more than 50 days. I had it again 2 weeks ago and again last night.

The only way I've found to quickly restart the node is to completely delete the directory containing the data.

And so, when that happens, I have to connect to the web interface of my VPS to restart it first because I no longer have access via ssh (I can't even ping the VPS) .

But it's really boring.

Have you ever had the same thing and above all, do you know how to prevent it?

(I use the 1.3.2 binary file that I downloaded, and it's an archive node)

May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Consumed 12.273s CPU time. May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Scheduled restart job, restart counter is at 61. May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: Stopped Node Ternoa. May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Consumed 12.273s CPU time. May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: Started Node Ternoa. May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 Ternoa Node May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 ✌️ version 1.1.1 May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 ❤️ by Ternoa, 2020-2024 May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 📋 Chain specification: Ternoa Mainnet May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 🏷 Node name: NodePainNodeGain May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 👤 Role: FULL May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 💾 Database: RocksDb at /blocks/chains/mainnet-live/db/full May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 ⛓ Native runtime: mainnet-21 (mainnet-0.tx1.au1) May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: ==================== May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: Version: 1.1.1 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 0: sp_panic_handler::set::{{closure}} May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 1: std::panicking::rust_panic_with_hook May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:702:17 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 2: std::panicking::begin_panic_handler::{{closure}} May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:588:13 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 3: std::sys_common::backtrace::__rust_end_short_backtrace May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/sys_common/backtrace.rs:138:18 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 4: rust_begin_unwind May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:584:5 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 5: core::panicking::panic_fmt May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/core/src/panicking.rs:142:14 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 6: core::result::unwrap_failed May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/core/src/result.rs:1814:5 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 8: <&mut dyn sp_externalities::Externalities as sp_io::storage::Storage>::get_version_1 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 9: sp_externalities::scope_limited::with_externalities May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 10: tracing::span::Span::in_scope May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 11: sp_io::storage::get_version_1 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 12: sp_io::storage::ExtStorageGetVersion1::call May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 13: <sc_executor_wasmtime::imports::Registry as sp_wasm_interface::HostFunctionRegistry>::with_function_context May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 14: std::panicking::try May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 15: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1),R>>::into_func::wasm_to_host_shim May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 16: May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 17: May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 18: May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 19: May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 20: wasmtime_runtime::traphandlers::catch_traps::call_closure May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 21: wasmtime_setjmp May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 22: wasmtime_runtime::traphandlers::<impl wasmtime_runtime::traphandlers::call_thread_state::CallThreadState>::with May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 23: wasmtime_runtime::traphandlers::catch_traps May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 24: wasmtime::func::invoke_wasm_and_catch_traps May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 25: wasmtime::func::typed::TypedFunc<Params,Results>::call May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 26: sc_executor_wasmtime::instance_wrapper::EntryPoint::call May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 27: sc_executor_wasmtime::runtime::perform_call May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 28: <sc_executor_wasmtime::runtime::WasmtimeInstance as sc_executor_common::wasm_runtime::WasmInstance>::call_with_allocation_stats May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 29: sc_executor_common::wasm_runtime::WasmInstance::call_export May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 30: std::panicking::try May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 31: std::thread::local::LocalKey<T>::with May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 32: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}} May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 33: sc_executor::wasm_runtime::RuntimeCache::with_instance May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 34: <sc_executor::native_executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 35: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 36: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 37: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 38: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 39: <mainnet_runtime::RuntimeApiImpl<SR_API_BLOCK,RuntimeApiImplCall> as sp_api::Core<SR_API_BLOCK>>::__runtime_api_internal_call_api_at May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 40: sp_consensus_babe::BabeApi::configuration May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 41: ternoa_service::new_partial May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 42: ternoa_service::new_full May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 43: tokio::runtime::park::CachedParkThread::block_on May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 44: tokio::runtime::scheduler::multi_thread::MultiThread::block_on May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 45: tokio::runtime::runtime::Runtime::block_on May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 46: sc_cli::runner::Runner<C>::run_node_until_exit May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 47: ternoa_cli::command::run May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 48: ternoa::main May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 49: std::sys_common::backtrace::__rust_begin_short_backtrace May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 50: std::rt::lang_start::{{closure}} May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 51: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/core/src/ops/function.rs:280:13 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try::do_call May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:492:40 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:456:19 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panic::catch_unwind May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panic.rs:137:14 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::rt::lang_start_internal::{{closure}} May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/rt.rs:128:48 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try::do_call May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:492:40 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:456:19 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panic::catch_unwind May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panic.rs:137:14 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::rt::lang_start_internal May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/rt.rs:128:20 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 52: main May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 53: May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 54: __libc_start_main May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 55: _start May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: Thread 'main' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0xce06c4bff55e0fef101af3ed89f4bbda49a9b459f3d9f598fd18f28c9498c16a"', /home/peshwar9/.cargo/git/checkouts/substrate-7e08433d4c370a21/a3ed011/primitives/state-machine/src/ext.rs:189 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: This is a bug. Please report it at: May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: https://github.com/capsule-corp-ternoa/chain/issues May 12 10:21:28 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Main process exited, code=killed, status=11/SEGV May 12 10:21:28 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Failed with result 'signal'.