Bug: panic at missing header

[bleetube]

Describe the bug This morning I had to hard reset the x86_64 archlinux machine running bitcoind. Since then, electrs 0.9.6 fails start due to a missing header. I'm guessing I need to rebuild my electrum database from scratch, so this is nominally just a copy of my panic log. bitcoind seems to be fine, as it still syncs to the latest block on startup. Feel free to close this unless you want more information or tests performed.

# doas -u bitcoin /opt/bin/electrs --conf /opt/electrs/config.toml
Starting electrs 0.9.6 on x86_64 linux with Config { network: Bitcoin, db_path: "/opt/electrs/db/bitcoin", daemon_dir: "/opt/bitcoind", daemon_auth: CookieFile("/opt/bitcoind/.cookie"), daemon_rpc_addr: 127.0.0.1:8332, daemon_p2p_addr: 127.0.0.1:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: None, reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, disable_electrum_rpc: false, server_banner: "Welcome to electrs 0.9.6 (Electrum Rust Server)!", args: [] }
thread 'main' panicked at 'missing header 0000000000000000000161ff595044fa214f2f908ab80626a7c5a059fd036f55 while loading from DB', src/chain.rs:79:25
stack backtrace:
   0:     0x55a2077d468c - std::backtrace_rs::backtrace::libunwind::trace::h11dc6469a6e52543
                               at /rustc/1.59.0/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x55a2077d468c - std::backtrace_rs::backtrace::trace_unsynchronized::hd4036938d0c3ae29
                               at /rustc/1.59.0/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55a2077d468c - std::sys_common::backtrace::_print_fmt::hf909b4c4e1107d0b
                               at /rustc/1.59.0/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x55a2077d468c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h4becf6587ad7aa1d
                               at /rustc/1.59.0/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x55a2077fa15c - core::fmt::write::h557b7f443eab4320
                               at /rustc/1.59.0/library/core/src/fmt/mod.rs:1168:17
   5:     0x55a2077cc6c5 - std::io::Write::write_fmt::h489ea24feb4a5ea4
                               at /rustc/1.59.0/library/std/src/io/mod.rs:1660:15
   6:     0x55a2077d6d70 - std::sys_common::backtrace::_print::h2e67187980d82d1e
                               at /rustc/1.59.0/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x55a2077d6d70 - std::sys_common::backtrace::print::ha1439b6972e85b57
                               at /rustc/1.59.0/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x55a2077d6d70 - std::panicking::default_hook::{{closure}}::hfa0b382729055e14
                               at /rustc/1.59.0/library/std/src/panicking.rs:211:50
   9:     0x55a2077d6929 - std::panicking::default_hook::h41128bfd08e307a0
                               at /rustc/1.59.0/library/std/src/panicking.rs:228:9
  10:     0x55a2077d73a6 - std::panicking::rust_panic_with_hook::h9e4813ea689fc43b
                               at /rustc/1.59.0/library/std/src/panicking.rs:606:17
  11:     0x55a2077d7100 - std::panicking::begin_panic_handler::{{closure}}::h522a2eaaa50f1034
                               at /rustc/1.59.0/library/std/src/panicking.rs:502:13
  12:     0x55a2077d4b24 - std::sys_common::backtrace::__rust_end_short_backtrace::h9cefd2b43c0486be
                               at /rustc/1.59.0/library/std/src/sys_common/backtrace.rs:139:18
  13:     0x55a2077d6e69 - rust_begin_unwind
                               at /rustc/1.59.0/library/std/src/panicking.rs:498:5
  14:     0x55a2071ccc81 - core::panicking::panic_fmt::h8ef5ad8e70ca9bc8
                               at /rustc/1.59.0/library/core/src/panicking.rs:116:14
  15:     0x55a2071fc3ee - electrs::chain::Chain::load::hae83e4cb07f91d12
  16:     0x55a207210e4d - electrs::index::Index::load::ha973f6dd847fbdcb
  17:     0x55a2072774d3 - electrs::tracker::Tracker::new::hac2bb67302b543b5
  18:     0x55a2072560b4 - electrs::electrum::Rpc::new::h818a261de4d6e802
  19:     0x55a2071ecb27 - electrs::server::run::he228429b53a450a4
  20:     0x55a2071e62a3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h72ddd7080766914f
  21:     0x55a2071e61cd - std::rt::lang_start::{{closure}}::h7a67cc6083aff9fe
  22:     0x55a2077d3e8e - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hd1208b862f1fb9fd
                               at /rustc/1.59.0/library/core/src/ops/function.rs:259:13
  23:     0x55a2077d3e8e - std::panicking::try::do_call::hcc194fcbf1b91aa1
                               at /rustc/1.59.0/library/std/src/panicking.rs:406:40
  24:     0x55a2077d3e8e - std::panicking::try::ha62166a0d570ae0e
                               at /rustc/1.59.0/library/std/src/panicking.rs:370:19
  25:     0x55a2077d3e8e - std::panic::catch_unwind::hdbb49d505e2dff37
                               at /rustc/1.59.0/library/std/src/panic.rs:133:14
  26:     0x55a2077d3e8e - std::rt::lang_start_internal::{{closure}}::he1f6f465111b6703
                               at /rustc/1.59.0/library/std/src/rt.rs:128:48
  27:     0x55a2077d3e8e - std::panicking::try::do_call::h752b2472b65dc578
                               at /rustc/1.59.0/library/std/src/panicking.rs:406:40
  28:     0x55a2077d3e8e - std::panicking::try::h5e5e1df37906fd95
                               at /rustc/1.59.0/library/std/src/panicking.rs:370:19
  29:     0x55a2077d3e8e - std::panic::catch_unwind::h74042ea207983c3e
                               at /rustc/1.59.0/library/std/src/panic.rs:133:14
  30:     0x55a2077d3e8e - std::rt::lang_start_internal::h57d0ce2765e13f39
                               at /rustc/1.59.0/library/std/src/rt.rs:128:20
  31:     0x55a2071e6292 - main
  32:     0x7fa507710310 - __libc_start_call_main
  33:     0x7fa5077103c1 - __libc_start_main@GLIBC_2.2.5
  34:     0x55a2071e6025 - _start
  35:                0x0 - <unknown>

Electrs version 0.9.6

To Reproduce Not sure this can be reliably reproduced. Steps to reproduce the behavior:

1. something bad with systemd such that it stops replying to rpc commands
2. pkill bitcoind && reboot -f
3. Wait
4. See error

System running electrs

• Deployment method: manual, iirc built with cargo
• Archlinux, kernel 5.17.2

Electrum client mempool-backend, which is disabled for now and not causing the panic.

[Kixunil]

Thanks for the report! I think it'd be useful to handle something like this more gracefully.

[bleetube]

just fyi this machine no longer passes memtest so the source of the error turned out to be faulty ram.