stacks-core
stacks-core copied to clipboard
Published
20 hours ago •
stacks-network
stacks-network
bitcoin reorg test failures
trafficstars
tracking here vs in #4601
https://github.com/stacks-network/stacks-core/blob/next/stackslib/src/burnchains/affirmation.rs#L791
using latest commit on next as of the time of this issue and running locally with bitcoin set to log to /tmp/bitcoind.log:
bitcoind spawn: "bitcoind" "-regtest" "-debuglogfile=/tmp/bitcoind.log" "-rest" "-txindex=1" "-server=1" "-listenonion=0" "-rpcbind=127.0.0.1" "-port=8333" "-datadir=/tmp/stacks-node-tests/integrations-neon/b7e3fee19e7c3a71-1712171722/neon/burnchain" "-rpcport=8332" "-rpcuser=neon-tester" "-rpcpassword=neon-tester-pass"
bitcoind startup finished
thread 'burnchain-db' panicked at stackslib/src/burnchains/affirmation.rs:791:17:
assertion failed: *last_vtxindex < opdata.vtxindex
stack backtrace:
0: 0x558affc482e6 - std::backtrace_rs::backtrace::libunwind::trace::hbee8a7973eeb6c93
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
1: 0x558affc482e6 - std::backtrace_rs::backtrace::trace_unsynchronized::hc8ac75eea3aa6899
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x558affc482e6 - std::sys_common::backtrace::_print_fmt::hc7f3e3b5298b1083
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
3: 0x558affc482e6 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbb235daedd7c6190
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
4: 0x558affc7a560 - core::fmt::rt::Argument::fmt::h76c38a80d925a410
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
5: 0x558affc7a560 - core::fmt::write::h3ed6aeaa977c8e45
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
6: 0x558affc43cbf - std::io::Write::write_fmt::h78b18af5775fedb5
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
7: 0x558affc480c4 - std::sys_common::backtrace::_print::h5d645a07e0fcfdbb
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
8: 0x558affc480c4 - std::sys_common::backtrace::print::h85035a511aafe7a8
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
9: 0x558affc49d37 - std::panicking::default_hook::{{closure}}::hcce8cea212785a25
10: 0x558affc49a99 - std::panicking::default_hook::hf5fcb0f213fe709a
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
11: 0x558affc4a1c8 - std::panicking::rust_panic_with_hook::h095fccf1dc9379ee
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:779:13
12: 0x558affc4a069 - std::panicking::begin_panic_handler::{{closure}}::h032ba12139b353db
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:649:13
13: 0x558affc487e6 - std::sys_common::backtrace::__rust_end_short_backtrace::h9259bc2ff8fd0f76
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
14: 0x558affc49e00 - rust_begin_unwind
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
15: 0x558afe206ee5 - core::panicking::panic_fmt::h784f20a50eaab275
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
16: 0x558afe206fa3 - core::panicking::panic::hb837a5ebbbe5b188
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:144:5
17: 0x558afeebb1ca - blockstack_lib::burnchains::affirmation::inner_find_heaviest_block_commit_ptr::ha36fbe5ec855d348
at /home/wileyj/stacks-core/stackslib/src/burnchains/affirmation.rs:791:17
18: 0x558afe5fa878 - blockstack_lib::burnchains::affirmation::find_heaviest_block_commit::h622ab1c14098f878
at /home/wileyj/stacks-core/stackslib/src/burnchains/affirmation.rs:933:15
19: 0x558afe5fa878 - blockstack_lib::burnchains::affirmation::find_pox_anchor_block::h97cc716b670d6c45
at /home/wileyj/stacks-core/stackslib/src/burnchains/affirmation.rs:1110:44
20: 0x558afe5fa878 - blockstack_lib::burnchains::affirmation::update_pox_affirmation_maps::h82546d96f294b5e2
at /home/wileyj/stacks-core/stackslib/src/burnchains/affirmation.rs:1165:9
21: 0x558afe20d947 - blockstack_lib::burnchains::burnchain::<impl blockstack_lib::burnchains::Burnchain>::process_affirmation_maps::h2d9b60b24ce5d777
at /home/wileyj/stacks-core/stackslib/src/burnchains/burnchain.rs:992:13
22: 0x558afe20d947 - blockstack_lib::burnchains::burnchain::<impl blockstack_lib::burnchains::Burnchain>::process_block::h51d9ea280e6399ec
at /home/wileyj/stacks-core/stackslib/src/burnchains/burnchain.rs:955:9
23: 0x558afe226ead - blockstack_lib::burnchains::burnchain::<impl blockstack_lib::burnchains::Burnchain>::sync_with_indexer::{{closure}}::hfe48dea4f2c8392f
at /home/wileyj/stacks-core/stackslib/src/burnchains/burnchain.rs:1602:42
24: 0x558afe226ead - std::sys_common::backtrace::__rust_begin_short_backtrace::h81ef0ad0721fe936
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:155:18
25: 0x558afe4653c6 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hb8685abce99751e0
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:529:17
26: 0x558afe4653c6 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h891c11d856ddc008
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
27: 0x558afe4653c6 - std::panicking::try::do_call::h02bc9c0709b2847e
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
28: 0x558afe4653c6 - std::panicking::try::h6ae156383a800ed9
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
29: 0x558afe4653c6 - std::panic::catch_unwind::h56b55f3050c27009
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
30: 0x558afe4653c6 - std::thread::Builder::spawn_unchecked_::{{closure}}::h3c42cb7a1d383546
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:528:30
31: 0x558afe4653c6 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hdab23c66752d11f1
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
32: 0x558affc51425 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h12de4fc57affb195
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
33: 0x558affc51425 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h3c619f45059d5cf1
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
34: 0x558affc51425 - std::sys::unix::thread::Thread::new::thread_start::hbac657605e4b7389
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
35: 0x7f7225e1e134 - <unknown>
36: 0x7f7225e9e7dc - <unknown>
37: 0x0 - <unknown>
thread '<unnamed>' panicked at /home/wileyj/stacks-core/stackslib/src/burnchains/burnchain.rs:1654:51:
called `Result::unwrap()` on an `Err` value: Any { .. }
stack backtrace:
0: 0x558affc482e6 - std::backtrace_rs::backtrace::libunwind::trace::hbee8a7973eeb6c93
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
1: 0x558affc482e6 - std::backtrace_rs::backtrace::trace_unsynchronized::hc8ac75eea3aa6899
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x558affc482e6 - std::sys_common::backtrace::_print_fmt::hc7f3e3b5298b1083
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
3: 0x558affc482e6 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbb235daedd7c6190
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
4: 0x558affc7a560 - core::fmt::rt::Argument::fmt::h76c38a80d925a410
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
5: 0x558affc7a560 - core::fmt::write::h3ed6aeaa977c8e45
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
6: 0x558affc43cbf - std::io::Write::write_fmt::h78b18af5775fedb5
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
7: 0x558affc480c4 - std::sys_common::backtrace::_print::h5d645a07e0fcfdbb
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
8: 0x558affc480c4 - std::sys_common::backtrace::print::h85035a511aafe7a8
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
9: 0x558affc49d37 - std::panicking::default_hook::{{closure}}::hcce8cea212785a25
10: 0x558affc49a99 - std::panicking::default_hook::hf5fcb0f213fe709a
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
11: 0x558affc4a1c8 - std::panicking::rust_panic_with_hook::h095fccf1dc9379ee
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:779:13
12: 0x558affc4a0a2 - std::panicking::begin_panic_handler::{{closure}}::h032ba12139b353db
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
13: 0x558affc487e6 - std::sys_common::backtrace::__rust_end_short_backtrace::h9259bc2ff8fd0f76
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
14: 0x558affc49e00 - rust_begin_unwind
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
15: 0x558afe206ee5 - core::panicking::panic_fmt::h784f20a50eaab275
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
16: 0x558afe207523 - core::result::unwrap_failed::h03d8a5018196e1cd
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1649:5
17: 0x558afe218116 - core::result::Result<T,E>::unwrap::hf67852d0f50c2827
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1073:23
18: 0x558afe210f64 - blockstack_lib::burnchains::burnchain::<impl blockstack_lib::burnchains::Burnchain>::sync_with_indexer::h669b7d8362ad6014
at /home/wileyj/stacks-core/stackslib/src/burnchains/burnchain.rs:1654:34
19: 0x558afe474bb8 - stacks_node::burnchains::bitcoin_regtest_controller::BitcoinRegtestController::receive_blocks::h8ee43b25e7d65f19
at /home/wileyj/stacks-core/testnet/stacks-node/src/burnchains/bitcoin_regtest_controller.rs:508:19
20: 0x558afe4838a3 - <stacks_node::burnchains::bitcoin_regtest_controller::BitcoinRegtestController as stacks_node::burnchains::BurnchainController>::sync::h1163a5fd0101c5aa
21: 0x558afe406e41 - stacks_node::run_loop::neon::RunLoop::start::heeb507e00e66fcad
at /home/wileyj/stacks-core/testnet/stacks-node/src/run_loop/neon.rs:1200:27
22: 0x558afe2229a8 - stacks_node::tests::neon_integrations::bitcoin_reorg_flap::{{closure}}::h7e465e7d453dcf90
at /home/wileyj/stacks-core/testnet/stacks-node/src/tests/neon_integrations.rs:12177:27
23: 0x558afe2229a8 - std::sys_common::backtrace::__rust_begin_short_backtrace::h0ed6c9583c071abd
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:155:18
24: 0x558afe4526ec - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h543c97dc0350987d
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:529:17
25: 0x558afe4526ec - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h8e352ec8ec7d8511
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
26: 0x558afe4526ec - std::panicking::try::do_call::he538037d84e40ea6
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
27: 0x558afe4526ec - std::panicking::try::h1dd97a020770998d
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
28: 0x558afe4526ec - std::panic::catch_unwind::hf4ac83249555294f
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
29: 0x558afe4526ec - std::thread::Builder::spawn_unchecked_::{{closure}}::h3e6aaf1211d4c2c8
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:528:30
30: 0x558afe4526ec - core::ops::function::FnOnce::call_once{{vtable.shim}}::h06b3e5fed3540602
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
31: 0x558affc51425 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h12de4fc57affb195
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
32: 0x558affc51425 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h3c619f45059d5cf1
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
33: 0x558affc51425 - std::sys::unix::thread::Thread::new::thread_start::hbac657605e4b7389
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
34: 0x7f7225e1e134 - <unknown>
35: 0x7f7225e9e7dc - <unknown>
36: 0x0 - <unknown>
thread 'tests::neon_integrations::bitcoin_reorg_flap' panicked at testnet/stacks-node/src/tests/neon_integrations.rs:12258:5:
assertion `left == right` failed
left: 220
right: 225
stack backtrace:
0: 0x558affc482e6 - std::backtrace_rs::backtrace::libunwind::trace::hbee8a7973eeb6c93
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
1: 0x558affc482e6 - std::backtrace_rs::backtrace::trace_unsynchronized::hc8ac75eea3aa6899
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x558affc482e6 - std::sys_common::backtrace::_print_fmt::hc7f3e3b5298b1083
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
3: 0x558affc482e6 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbb235daedd7c6190
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
4: 0x558affc7a560 - core::fmt::rt::Argument::fmt::h76c38a80d925a410
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
5: 0x558affc7a560 - core::fmt::write::h3ed6aeaa977c8e45
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
6: 0x558affc43cbf - std::io::Write::write_fmt::h78b18af5775fedb5
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
7: 0x558affc480c4 - std::sys_common::backtrace::_print::h5d645a07e0fcfdbb
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
8: 0x558affc480c4 - std::sys_common::backtrace::print::h85035a511aafe7a8
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
9: 0x558affc49d37 - std::panicking::default_hook::{{closure}}::hcce8cea212785a25
10: 0x558affc49a99 - std::panicking::default_hook::hf5fcb0f213fe709a
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
11: 0x558affc4a1c8 - std::panicking::rust_panic_with_hook::h095fccf1dc9379ee
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:779:13
12: 0x558affc4a0a2 - std::panicking::begin_panic_handler::{{closure}}::h032ba12139b353db
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
13: 0x558affc487e6 - std::sys_common::backtrace::__rust_end_short_backtrace::h9259bc2ff8fd0f76
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
14: 0x558affc49e00 - rust_begin_unwind
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
15: 0x558afe206ee5 - core::panicking::panic_fmt::h784f20a50eaab275
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
16: 0x558afe2073bb - core::panicking::assert_failed_inner::hbf94b40c37b92af0
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:342:17
17: 0x558afe17b5e2 - core::panicking::assert_failed::h0fd1819780793ad9
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:297:5
18: 0x558afe3cf4ce - stacks_node::tests::neon_integrations::bitcoin_reorg_flap::h882d3761e89b9c41
at /home/wileyj/stacks-core/testnet/stacks-node/src/tests/neon_integrations.rs:12258:5
19: 0x558afe2d7a79 - stacks_node::tests::neon_integrations::bitcoin_reorg_flap::{{closure}}::heae1dc136f301ca3
at /home/wileyj/stacks-core/testnet/stacks-node/src/tests/neon_integrations.rs:12154:24
20: 0x558afe2d7a79 - core::ops::function::FnOnce::call_once::h33c732e261d110f0
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
21: 0x558afe81965f - core::ops::function::FnOnce::call_once::h8dc6907944022cf6
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
22: 0x558afe81965f - test::__rust_begin_short_backtrace::haae1a87433f1efb3
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/test/src/lib.rs:627:18
23: 0x558afe8183e1 - test::run_test_in_process::{{closure}}::h8c7decfa7c14e152
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/test/src/lib.rs:650:60
24: 0x558afe8183e1 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h19e6ff056d9d21e9
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
25: 0x558afe8183e1 - std::panicking::try::do_call::h89c848fcaa37c035
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
26: 0x558afe8183e1 - std::panicking::try::h57ab3dc74e2839b8
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
27: 0x558afe8183e1 - std::panic::catch_unwind::hfb6a1b1abc120fb9
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
28: 0x558afe8183e1 - test::run_test_in_process::h5ae2f9875edd562d
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/test/src/lib.rs:650:27
29: 0x558afe8183e1 - test::run_test::{{closure}}::h35d7300d8928a067
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/test/src/lib.rs:573:43
30: 0x558afe7df716 - test::run_test::{{closure}}::h7525ced405d23d1b
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/test/src/lib.rs:601:41
31: 0x558afe7df716 - std::sys_common::backtrace::__rust_begin_short_backtrace::h4e7db78ce05afad8
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:155:18
32: 0x558afe7e4777 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hcfbcb64f1a1b3482
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:529:17
33: 0x558afe7e4777 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h9d89c5c4108bd689
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
34: 0x558afe7e4777 - std::panicking::try::do_call::h8a4869bc94ec50c9
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
35: 0x558afe7e4777 - std::panicking::try::h9a576f20ff81ac30
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
36: 0x558afe7e4777 - std::panic::catch_unwind::hbcb4e3f860ef9830
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
37: 0x558afe7e4777 - std::thread::Builder::spawn_unchecked_::{{closure}}::h93c79a6be1505948
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:528:30
38: 0x558afe7e4777 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h426d96740c81bdaf
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
39: 0x558affc51425 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h12de4fc57affb195
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
40: 0x558affc51425 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h3c619f45059d5cf1
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
41: 0x558affc51425 - std::sys::unix::thread::Thread::new::thread_start::hbac657605e4b7389
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
42: 0x7f7225e1e134 - <unknown>
43: 0x7f7225e9e7dc - <unknown>
44: 0x0 - <unknown>
Canceling due to test failure
------------
Summary [ 674.141s] 1 test run: 0 passed, 1 failed, 4524 skipped
TRY 4 FAIL [ 168.327s] stacks-node::bin/stacks-node tests::neon_integrations::bitcoin_reorg_flap
error: test run failed
the pids are indeed being stopped
bitcoind log is showing it being shutdown as expected:
...
2024-04-03T19:14:42Z Saw new header hash=7dcec404fc1509328a55379cd82be74920cf81df45ed3faee8a825b1742f88f2 height=215
2024-04-03T19:14:42Z UpdateTip: new best=7dcec404fc1509328a55379cd82be74920cf81df45ed3faee8a825b1742f88f2 height=215 version=0x30000000 log2_work=8.754888 tx=225 date='2024-04-03T19:14:42Z' progress=1.000000 cache=0.0MiB(9txo)
2024-04-03T19:14:42Z [default wallet] AddToWallet e762a5e466dd0fa55a6d345c908671075a2d700d839ff7761ec20952967f7ffc new
2024-04-03T19:14:46Z [default wallet] AddToWallet 88f2d272a4de245354b949e117901018584fcf4138b9535eb0ee4c99a788f95b new
2024-04-03T19:14:47Z Shutdown: In progress...
2024-04-03T19:14:47Z addcon thread exit
2024-04-03T19:14:47Z opencon thread exit
2024-04-03T19:14:47Z net thread exit
2024-04-03T19:14:47Z msghand thread exit
2024-04-03T19:14:47Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat started
2024-04-03T19:14:47Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat completed (0.00s)
2024-04-03T19:14:47Z scheduler thread exit
2024-04-03T19:14:47Z Writing 1 unbroadcast transactions to disk.
2024-04-03T19:14:47Z Dumped mempool: 1.886e-06s to copy, 0.000984033s to dump
2024-04-03T19:14:47Z Bitcoin Core version v25.0.0 (release build)
2024-04-03T19:14:47Z Using the 'x86_shani(1way,2way)' SHA256 implementation
2024-04-03T19:14:47Z Using RdSeed as an additional entropy source
2024-04-03T19:14:47Z Using RdRand as an additional entropy source
...
i'm going to keep hacking at this locally before i attempt to run remotely, but it seems there's either a problem with the test assertion or the order of operations here.
Ok, I can reproduce it now locally too. This is a different problem from the one I looked at before, which I could not reproduce locally. I can dig into this.
diff --git a/testnet/stacks-node/src/tests/bitcoin_regtest.rs b/testnet/stacks-node/src/tests/bitcoin_regtest.rs
index 5f8b1aabd..eac1237cc 100644
--- a/testnet/stacks-node/src/tests/bitcoin_regtest.rs
+++ b/testnet/stacks-node/src/tests/bitcoin_regtest.rs
@@ -43,8 +43,7 @@ impl BitcoinCoreController {
command
.stdout(Stdio::piped())
.arg("-regtest")
- .arg("-nodebug")
- .arg("-nodebuglogfile")
+ .arg("-debuglogfile=/tmp/bitcoind.log")
.arg("-rest")
.arg("-txindex=1")
.arg("-server=1")
and to run the test, i'm using:
export BITCOIND_TEST=1
export RUST_BACKTRACE=full
cargo nextest run \
--verbose \
--archive-file ~/test_archive.tar.zst \
--test-threads 1 \
--retries 3 \
--run-ignored all \
--fail-fast \
--success-output immediate-final \
--status-level fail \
tests::neon_integrations::bitcoin_reorg_flap
Basically, the reorg flap caused was greater than the anchor threshold which ended up causing an assertion in the code to trigger. I'll fix that and then hopefully we are back to the original error (or no error at all 🤞).
I don't understand why the changes in https://github.com/stacks-network/stacks-core/pull/4601/commits/a4b5295c74509f5cfb9081aab2aa73bfb6a75791 don't have the same result as the previous changes. I just wanted to revert the changes to the defaults so that the other tests don't all need to change, then apply those specific values just for this test. It should behave the same way, but it is back to failing locally hitting the assertion. 🤔