Research segmentation fault in the subspace-node application.

Open shamil-gadelshin opened this issue 3 years ago • 2 comments

We found a segmentation fault in the subspace-node application. It happens only in the specific commit with the specific compiler version and the build command.

Conditions to reproduce

macOS Monterrey 12.3.1 Apple M1Max
build command: cargo build --package subspace-networking --package subspace-farmer --package subspace-node
run command: ./target/debug/subspace-node
branch: https://github.com/subspace/subspace/tree/segmentation-fault-backup-branch

It was found in one of the commits for this PR. Here are the details: https://github.com/subspace/subspace/pull/1212#issuecomment-1459606048

Backtrace:

* thread #10, name = 'tokio-runtime-worker', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x00000001c92d985c libunwind.dylib`libunwind::CFI_Parser<libunwind::LocalAddressSpace>::parseFDEInstructions(libunwind::LocalAddressSpace&, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::FDE_Info const&, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::CIE_Info const&, unsigned long, int, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::PrologInfo*) + 204
    frame #1: 0x00000001c92d9710 libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::getInfoFromFdeCie(libunwind::CFI_Parser<libunwind::LocalAddressSpace>::FDE_Info const&, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::CIE_Info const&, unsigned long, unsigned long) + 100
    frame #2: 0x00000001c92d93e8 libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::getInfoFromDwarfSection(unsigned long, libunwind::UnwindInfoSections const&, unsigned int) + 184
    frame #3: 0x00000001c92d92a0 libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::setInfoBasedOnIPRegister(bool) + 1012
    frame #4: 0x00000001c92db788 libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::step() + 696
    frame #5: 0x00000001c92de138 libunwind.dylib`_Unwind_Backtrace + 352
    frame #6: 0x0000000104728684 subspace-node`backtrace::backtrace::trace_unsynchronized::hfda9b107e91f41f0 [inlined] backtrace::backtrace::libunwind::trace::h4d69f540e3e59346(cb=&mut dyn core::ops::function::FnMut<(&backtrace::backtrace::Frame), Output=bool> @ 0x0000000171063fa0) at libunwind.rs:93:5
    frame #7: 0x0000000104728680 subspace-node`backtrace::backtrace::trace_unsynchronized::hfda9b107e91f41f0(cb=<unavailable>) at mod.rs:66:5
    frame #8: 0x0000000104728610 subspace-node`backtrace::backtrace::trace::h8287e8d81a9b688c(cb={closure_env#0} @ 0x0000000171064088) at mod.rs:53:14
    frame #9: 0x000000010471cb50 subspace-node`backtrace::capture::Backtrace::create::h4f4606a283baec1e(ip=4369533660) at capture.rs:176:9
    frame #10: 0x000000010471caf0 subspace-node`backtrace::capture::Backtrace::new_unresolved::hb2c057ed913f1279 at capture.rs:170:9
    frame #11: 0x00000001019a56bc subspace-node`sc_utils::mpsc::inner::tracing_unbounded::h8ee4cbb9049b5ca3(name="mpsc_import_notification_stream", queue_size_warning=100000) at mpsc.rs:111:33
    frame #12: 0x00000001006a20f4 subspace-node`_$LT$sc_service..client..client..Client$LT$B$C$E$C$Block$C$RA$GT$$u20$as$u20$sc_client_api..client..BlockchainEvents$LT$Block$GT$$GT$::import_notification_stream::hf367da8da2cbf634(self=0x0000000113809610) at client.rs:2000:24
    frame #13: 0x00000001010d5d78 subspace-node`sc_offchain::notification_future::_$u7b$$u7b$closure$u7d$$u7d$::h4baa3aa42d1149c2((null)=ResumeTy @ 0x0000000171064680) at lib.rs:218:2
    frame #14: 0x0000000100712e48 subspace-node`_$LT$sc_service..task_manager..prometheus_future..PrometheusFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h0b263a0dc9c6d97e(self=Pin<&mut sc_service::task_manager::prometheus_future::PrometheusFuture<sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>> @ 0x0000000171064720, cx=0x0000000171065780) at prometheus_future.rs:61:3
    frame #15: 0x0000000101c92394 subspace-node`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hc503d0113f5703bd(self=Pin<&mut core::panic::unwind_safe::AssertUnwindSafe<sc_service::task_manager::prometheus_future::PrometheusFuture<sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>> @ 0x0000000171064758, cx=0x0000000171065780) at unwind_safe.rs:296:9
    frame #16: 0x00000001004ce978 subspace-node`_$LT$futures_util..future..future..catch_unwind..CatchUnwind$LT$Fut$GT$$u20$as$u20$core..future..future..Future$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::h9f21a6787dee968e at catch_unwind.rs:36:42
    frame #17: 0x0000000101ca0968 subspace-node`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h996cfe3dca35931d(self=AssertUnwindSafe<futures_util::future::future::catch_unwind::{impl#1}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<sc_service::task_manager::prometheus_future::PrometheusFuture<sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>>> @ 0x00000001710647b0, _args=<unavailable>) at unwind_safe.rs:271:9
    frame #18: 0x0000000101694a48 subspace-node`std::panicking::try::do_call::h58778bf9224f56f8(data="\bd\U00000014\U00000013\U00000001") at panicking.rs:483:40
    frame #19: 0x000000010170a608 subspace-node`__rust_try + 32
    frame #20: 0x0000000101637394 subspace-node`std::panicking::try::h543cbea60d61bb1a(f=AssertUnwindSafe<futures_util::future::future::catch_unwind::{impl#1}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<sc_service::task_manager::prometheus_future::PrometheusFuture<sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>>> @ 0x00000001710648b8) at panicking.rs:447:19
    frame #21: 0x0000000101478b58 subspace-node`std::panic::catch_unwind::h470d7b6fc9f05415(f=AssertUnwindSafe<futures_util::future::future::catch_unwind::{impl#1}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<sc_service::task_manager::prometheus_future::PrometheusFuture<sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>>> @ 0x0000000171064910) at panic.rs:140:14
    frame #22: 0x00000001004cd1b8 subspace-node`_$LT$futures_util..future..future..catch_unwind..CatchUnwind$LT$Fut$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h6eefa797295eb88f(self=Pin<&mut futures_util::future::future::catch_unwind::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<sc_service::task_manager::prometheus_future::PrometheusFuture<sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>>> @ 0x0000000171064998, cx=0x0000000171065780) at catch_unwind.rs:36:9
    frame #23: 0x00000001012c65a4 subspace-node`_$LT$core..pin..Pin$LT$P$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h367d43eaa7fe429a(self=Pin<&mut core::pin::Pin<&mut futures_util::future::future::catch_unwind::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<sc_service::task_manager::prometheus_future::PrometheusFuture<sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>>>> @ 0x00000001710649f8, cx=0x0000000171065780) at future.rs:125:9
    frame #24: 0x00000001012ac9bc subspace-node`futures_util::future::future::FutureExt::poll_unpin::h63cca38c09cb72ee(self=0x0000000171064ac0, cx=0x0000000171065780) at mod.rs:562:9
    frame #25: 0x00000001012de94c subspace-node`_$LT$futures_util..future..select..Select$LT$A$C$B$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h26ba2f744c17e2dd(self=Pin<&mut futures_util::future::select::Select<exit_future::Exit, core::pin::Pin<&mut futures_util::future::future::catch_unwind::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<sc_service::task_manager::prometheus_future::PrometheusFuture<sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>>>>> @ 0x0000000171064aa8, cx=0x0000000171065780) at select.rs:108:35
    frame #26: 0x000000010092e298 subspace-node`sc_service::task_manager::SpawnTaskHandle::spawn_inner::_$u7b$$u7b$closure$u7d$$u7d$::hb795be859978d6fd((null)=ResumeTy @ 0x00000001710654d0) at mod.rs:168:32
    frame #27: 0x00000001015d1878 subspace-node`_$LT$tracing_futures..Instrumented$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h22df60c661a96ca1(self=Pin<&mut tracing_futures::Instrumented<sc_service::task_manager::{impl#2}::spawn_inner::{async_block_env#0}<core::option::Option<&str>, sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>> @ 0x0000000171065598, cx=0x0000000171065780) at lib.rs:283:9
    frame #28: 0x00000001013bd6d4 subspace-node`tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::hf1b1814942f195eb(ptr=0x0000000113146230) at core.rs:223:17
    frame #29: 0x00000001007b7354 subspace-node`tokio::loom::std::unsafe_cell::UnsafeCell$LT$T$GT$::with_mut::h8a44289f3a30f661(self=0x0000000113146230, f={closure_env#0}<tracing_futures::Instrumented<sc_service::task_manager::{impl#2}::spawn_inner::{async_block_env#0}<core::option::Option<&str>, sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>> @ 0x0000000171065738) at unsafe_cell.rs:14:9
    frame #30: 0x000000010139fd7c subspace-node`tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::h7be8e7bcc55d1485(self=0x0000000113146220, cx=Context @ 0x0000000171065780) at core.rs:212:13
    frame #31: 0x0000000101a2c214 subspace-node`tokio::runtime::task::harness::poll_future::_$u7b$$u7b$closure$u7d$$u7d$::hfcf6490cd2795c38 at harness.rs:476:19
    frame #32: 0x0000000101ca840c subspace-node`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hedb32e6988b677e7(self=AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing_futures::Instrumented<sc_service::task_manager::{impl#2}::spawn_inner::{async_block_env#0}<core::option::Option<&str>, sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>> @ 0x0000000171065820, _args=<unavailable>) at unwind_safe.rs:271:9
    frame #33: 0x00000001016c55e8 subspace-node`std::panicking::try::do_call::hff6cba35d7d57b27(data="\xb0Y\U00000006q\U00000001") at panicking.rs:483:40
    frame #34: 0x000000010170a608 subspace-node`__rust_try + 32
    frame #35: 0x00000001016289fc subspace-node`std::panicking::try::h2def57b3cf465af6(f=AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing_futures::Instrumented<sc_service::task_manager::{impl#2}::spawn_inner::{async_block_env#0}<core::option::Option<&str>, sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>> @ 0x0000000171065928) at panicking.rs:447:19
    frame #36: 0x000000010148002c subspace-node`std::panic::catch_unwind::h98bd2042104ab021(f=AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing_futures::Instrumented<sc_service::task_manager::{impl#2}::spawn_inner::{async_block_env#0}<core::option::Option<&str>, sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>> @ 0x0000000171065980) at panic.rs:140:14
    frame #37: 0x0000000101a03420 subspace-node`tokio::runtime::task::harness::poll_future::ha53644e8d3874c02(core=0x0000000113146220, cx=Context @ 0x0000000171065aa0) at harness.rs:464:18
    frame #38: 0x0000000101a30580 subspace-node`tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll_inner::h059d42c73ed95615(self=0x0000000171065b68) at harness.rs:198:27
    frame #39: 0x0000000101a8f61c subspace-node`tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::he0cdd2fe0f246f24(self=Harness<tracing_futures::Instrumented<sc_service::task_manager::{impl#2}::spawn_inner::{async_block_env#0}<core::option::Option<&str>, sc_offchain::notification_future::{async_fn_env#0}<sc_service::client::client::Client<sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_service::client::call_executor::LocalCallExecutor<subspace_runtime_primitives::opaque::Block, sc_client_db::Backend<subspace_runtime_primitives::opaque::Block>, sc_executor::native_executor::NativeElseWasmExecutor<subspace_node::ExecutorDispatch>>, subspace_runtime_primitives::opaque::Block, subspace_runtime::RuntimeApi>, subspace_runtime_primitives::opaque::Block, sc_service::task_manager::SpawnTaskHandle>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>> @ 0x0000000171065b68) at harness.rs:152:15
    frame #40: 0x0000000100497514 subspace-node`tokio::runtime::task::raw::poll::h192bcd97644e91e9(ptr=NonNull<tokio::runtime::task::core::Header> @ 0x0000000171065b90) at raw.rs:255:5
    frame #41: 0x0000000104c0b9e4 subspace-node`tokio::runtime::task::raw::RawTask::poll::h489de6e8ee578fc7(self=RawTask @ 0x0000000171065bb0) at raw.rs:200:18
    frame #42: 0x0000000104c23230 subspace-node`tokio::runtime::task::LocalNotified$LT$S$GT$::run::he19d39eb60a5ce83(self=LocalNotified<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>> @ 0x0000000171065be0) at mod.rs:394:9
    frame #43: 0x0000000104bffc4c subspace-node`tokio::runtime::scheduler::multi_thread::worker::Context::run_task::_$u7b$$u7b$closure$u7d$$u7d$::h84e36dccf94d15eb at worker.rs:464:13
    frame #44: 0x0000000104bffb50 subspace-node`tokio::runtime::scheduler::multi_thread::worker::Context::run_task::hcd6f5eec412bc09f at coop.rs:102:5
    frame #45: 0x0000000104bffaf0 subspace-node`tokio::runtime::scheduler::multi_thread::worker::Context::run_task::hcd6f5eec412bc09f [inlined] tokio::runtime::coop::budget::h6ce8c8bbdcddbd51(f={closure_env#0} @ 0x0000000171065df8) at coop.rs:68:5
    frame #46: 0x0000000104bffa7c subspace-node`tokio::runtime::scheduler::multi_thread::worker::Context::run_task::hcd6f5eec412bc09f(self=0x0000000171066138, task=Notified<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>> @ 0x0000000171065da0, core=0x0000600000c01c50) at worker.rs:463:9
    frame #47: 0x0000000104bff4c4 subspace-node`tokio::runtime::scheduler::multi_thread::worker::Context::run::h35f115b45a0211d9(self=0x0000000171066138, core=0x0000600000c01c50) at worker.rs:426:24
    frame #48: 0x0000000104bff31c subspace-node`tokio::runtime::scheduler::multi_thread::worker::run::_$u7b$$u7b$closure$u7d$$u7d$::h953a3052a24bd42d at worker.rs:406:17
    frame #49: 0x0000000104becd6c subspace-node`tokio::macros::scoped_tls::ScopedKey$LT$T$GT$::set::ha1df9a8b749d3658(self=0x0000000107a68d28, t=0x0000000171066138, f={closure_env#0} @ 0x0000000171066050) at scoped_tls.rs:61:9
    frame #50: 0x0000000104bff1d8 subspace-node`tokio::runtime::scheduler::multi_thread::worker::run::hec7b848051406546(worker=strong=1, weak=0) at worker.rs:403:5
    frame #51: 0x0000000104bff004 subspace-node`tokio::runtime::scheduler::multi_thread::worker::Launch::launch::_$u7b$$u7b$closure$u7d$$u7d$::hdf199494fc512da2 at worker.rs:365:45
    frame #52: 0x0000000104ba6e64 subspace-node`_$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h7e18deb8c880b44a(self=Pin<&mut tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>> @ 0x00000001710661c0, _cx=0x0000000171066320) at task.rs:42:21
    frame #53: 0x0000000104ba58ec subspace-node`tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::h3a532fe4c883a021(ptr=0x0000600002904028) at core.rs:223:17
    frame #54: 0x0000000104bc162c subspace-node`tokio::loom::std::unsafe_cell::UnsafeCell$LT$T$GT$::with_mut::hdc76fe27aacf2a5d(self=0x0000600002904028, f={closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> @ 0x00000001710662d8) at unsafe_cell.rs:14:9
    frame #55: 0x0000000104ba5750 subspace-node`tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::h6ca050bdeaeb9778(self=0x0000600002904020, cx=Context @ 0x0000000171066320) at core.rs:212:13
    frame #56: 0x0000000104c0b5f8 subspace-node`tokio::runtime::task::harness::poll_future::_$u7b$$u7b$closure$u7d$$u7d$::hf0ad68e28d044488 at harness.rs:476:19
    frame #57: 0x0000000104c1f5dc subspace-node`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h8b7cff2222466ccb(self=AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>> @ 0x00000001710663c0, _args=<unavailable>) at unwind_safe.rs:271:9
    frame #58: 0x0000000104bdbed4 subspace-node`std::panicking::try::do_call::h4cde5beff66d16cf(data="Pe\U00000006q\U00000001") at panicking.rs:483:40
    frame #59: 0x0000000104bdef34 subspace-node`__rust_try + 32
    frame #60: 0x0000000104bdae54 subspace-node`std::panicking::try::h2187f2802b351020(f=AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>> @ 0x00000001710664c8) at panicking.rs:447:19
    frame #61: 0x0000000104be780c subspace-node`std::panic::catch_unwind::h3aaa47be805fe8ba(f=AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>> @ 0x0000000171066520) at panic.rs:140:14
    frame #62: 0x0000000104c0b0c8 subspace-node`tokio::runtime::task::harness::poll_future::h0eb6fe283de50163(core=0x0000600002904020, cx=Context @ 0x0000000171066640) at harness.rs:464:18
    frame #63: 0x0000000104c09f9c subspace-node`tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll_inner::hb8abe1f07dd76c4a(self=0x0000000171066708) at harness.rs:198:27
    frame #64: 0x0000000104c09dd0 subspace-node`tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::h064e2989c358d313(self=Harness<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> @ 0x0000000171066708) at harness.rs:152:15
    frame #65: 0x0000000104c0bb9c subspace-node`tokio::runtime::task::raw::poll::h21ff805ff1fc8b8d(ptr=NonNull<tokio::runtime::task::core::Header> @ 0x0000000171066730) at raw.rs:255:5
    frame #66: 0x0000000104c0b9e4 subspace-node`tokio::runtime::task::raw::RawTask::poll::h489de6e8ee578fc7(self=RawTask @ 0x0000000171066750) at raw.rs:200:18
    frame #67: 0x0000000104c232f4 subspace-node`tokio::runtime::task::UnownedTask$LT$S$GT$::run::hc380dbd0c2218854(self=UnownedTask<tokio::runtime::blocking::schedule::BlockingSchedule> @ 0x0000000171066790) at mod.rs:431:9
    frame #68: 0x0000000104c0c014 subspace-node`tokio::runtime::blocking::pool::Task::run::hf9183f31a6c36560(self=Task @ 0x00000001710667c0) at pool.rs:159:9
    frame #69: 0x0000000104c0e2c4 subspace-node`tokio::runtime::blocking::pool::Inner::run::h7e75c1169ed96519(self=0x00000001131042a0, worker_thread_id=8) at pool.rs:511:17
    frame #70: 0x0000000104c0da3c subspace-node`tokio::runtime::blocking::pool::Spawner::spawn_thread::_$u7b$$u7b$closure$u7d$$u7d$::h8d4c91de125b0a12 at pool.rs:469:13
    frame #71: 0x0000000104bb3928 subspace-node`std::sys_common::backtrace::__rust_begin_short_backtrace::h2e780126145c7fc6(f=<unavailable>) at backtrace.rs:121:18
    frame #72: 0x0000000104bf37fc subspace-node`std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h7b0f84ada8fee6c8 at mod.rs:558:17
    frame #73: 0x0000000104c1f738 subspace-node`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::he358dd3067116721(self=<unavailable>, _args=<unavailable>) at unwind_safe.rs:271:9
    frame #74: 0x0000000104bdbad8 subspace-node`std::panicking::try::do_call::h033a2ba8d73228f0(data="\U00000001") at panicking.rs:483:40
    frame #75: 0x0000000104bdef34 subspace-node`__rust_try + 32
    frame #76: 0x0000000104bdb67c subspace-node`std::panicking::try::hb71f508995c2f5e5(f=<unavailable>) at panicking.rs:447:19
    frame #77: 0x0000000104be7898 subspace-node`std::panic::catch_unwind::h798fc43573958d93(f=<unavailable>) at panic.rs:140:14
    frame #78: 0x0000000104bf3664 subspace-node`std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::ha9ac686bf4a1dd4d at mod.rs:557:30
    frame #79: 0x0000000104bf3b60 subspace-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hd359b498661cbb11((null)=0x0000600001701200, (null)=<unavailable>) at function.rs:250:5
    frame #80: 0x0000000104eacb84 subspace-node`std::sys::unix::thread::Thread::new::thread_start::h48c8d7bde5831c16 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h6e817a21afa38368 at boxed.rs:1988:9 [opt]
    frame #81: 0x0000000104eacb78 subspace-node`std::sys::unix::thread::Thread::new::thread_start::h48c8d7bde5831c16 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::ha7a7c99dd4f01d31 at boxed.rs:1988:9 [opt]
    frame #82: 0x0000000104eacb74 subspace-node`std::sys::unix::thread::Thread::new::thread_start::h48c8d7bde5831c16 at thread.rs:108:17 [opt]
    frame #83: 0x00000001be7ca26c libsystem_pthread.dylib`_pthread_start + 148

Mar 08 '23 13:03 shamil-gadelshin

What if instead of building 3 packages you build just one binary, does it crash the same way?:

cargo run --bin subspace-node

Mar 08 '23 13:03 nazar-pc

What if instead of building 3 packages you build just one binary, does it crash the same way?:
cargo run --bin subspace-node

I tried it earlier. No, it does not !! 😮

Mar 08 '23 14:03 shamil-gadelshin

Since we've never seen this again, and we've upgraded compilers and dependencies, I'm going to close this ticket.

If it happens again we can re-open or make another ticket.

Jul 02 '25 21:07 teor2345