aptos-core
aptos-core copied to clipboard
Published
20 hours ago •
aptos-labs
aptos-labs
[Bug] Attempt to self-host transactions stream service gives error "Failed to retrieve oldest block information: NewBlockEvent at or after version xxx"
trafficstars
🐛 Bug
Got this error when trying to self-host transaction stream service (that has no docs available)
panicked at ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs:489:39:
called `Result::unwrap()` on an `Err` value: ServiceUnavailable(Json(AptosError { message: \"Failed to retrieve oldest block information: NewBlockEvent at or after version 2605577512 not found.\", error_code: InternalError, vm_error_code: None }), None, None, None, None, None, None, None, None)
And here're the exact logs
details = """
panicked at ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs:489:39:
called `Result::unwrap()` on an `Err` value: ServiceUnavailable(Json(AptosError { message: \"Failed to retrieve oldest block information: NewBlockEvent at or after version 2605577512 not found.\", error_code: InternalError, vm_error_code: None }), None, None, None, None, None, None, None, None)
Stack backtrace:
0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:564:25
1: <T as core::convert::Into<U>>::into
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/convert/mod.rs:759:9
2: aptos_api::context::Context::get_latest_ledger_info_wrapped::{{closure}}
at /aptos/api/src/context.rs:223:28
3: core::result::Result<T,E>::map_err
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs:829:27
4: aptos_api::context::Context::get_latest_ledger_info_wrapped
at /aptos/api/src/context.rs:223:14
5: aptos_indexer_grpc_table_info::table_info_service::TableInfoService::get_highest_known_version::{{closure}}
at /aptos/ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs:488:24
6: aptos_indexer_grpc_table_info::table_info_service::TableInfoService::run::{{closure}}
at /aptos/ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs:89:67
7: aptos_indexer_grpc_table_info::runtime::bootstrap::{{closure}}
at /aptos/ecosystem/indexer-grpc/indexer-grpc-table-info/src/runtime.rs:109:22
8: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:331:17
9: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9
10: tokio::runtime::task::core::Core<T,S>::poll
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:320:30
11: tokio::runtime::task::harness::poll_future::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:500:19
12: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
13: std::panicking::try::do_call
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
14: std::panicking::try
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
15: std::panic::catch_unwind
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
16: tokio::runtime::task::harness::poll_future
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:488:18
17: tokio::runtime::task::harness::Harness<T,S>::poll_inner
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:209:27
18: tokio::runtime::task::harness::Harness<T,S>::poll
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:154:15
19: tokio::runtime::task::raw::RawTask::poll
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:201:18
20: tokio::runtime::task::LocalNotified<S>::run
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/mod.rs:436:9
21: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:598:18
22: tokio::runtime::coop::with_budget
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
23: tokio::runtime::coop::budget
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
24: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:597:9
25: tokio::runtime::scheduler::multi_thread::worker::Context::run
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:548:24
26: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:513:21
27: tokio::runtime::context::scoped::Scoped<T>::set
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/scoped.rs:40:9
28: tokio::runtime::context::set_scheduler::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:26
29: std::thread::local::LocalKey<T>::try_with
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:284:16
30: std::thread::local::LocalKey<T>::with
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:260:9
31: tokio::runtime::context::set_scheduler
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:17
32: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:508:9
33: tokio::runtime::context::runtime::enter_runtime
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
34: tokio::runtime::scheduler::multi_thread::worker::run
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:500:5
35: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:466:45
36: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/task.rs:42:21
37: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:331:17
38: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9
39: tokio::runtime::task::core::Core<T,S>::poll
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:320:30
40: tokio::runtime::task::harness::poll_future::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:500:19
41: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
42: std::panicking::try::do_call
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
43: std::panicking::try
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
44: std::panic::catch_unwind
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
45: tokio::runtime::task::harness::poll_future
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:488:18
46: tokio::runtime::task::harness::Harness<T,S>::poll_inner
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:209:27
47: tokio::runtime::task::harness::Harness<T,S>::poll
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:154:15
48: tokio::runtime::task::raw::poll
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:271:5
49: tokio::runtime::task::raw::RawTask::poll
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:201:18
50: tokio::runtime::task::UnownedTask<S>::run
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/mod.rs:473:9
51: tokio::runtime::blocking::pool::Task::run
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:160:9
52: tokio::runtime::blocking::pool::Inner::run
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:518:17
53: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:476:13
54: std::sys_common::backtrace::__rust_begin_short_backtrace
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
55: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/mod.rs:528:17
56: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
57: std::panicking::try::do_call
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
58: std::panicking::try
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
59: std::panic::catch_unwind
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
60: std::thread::Builder::spawn_unchecked_::{{closure}}
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/mod.rs:527:30
61: core::ops::function::FnOnce::call_once{{vtable.shim}}
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
62: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2020:9
63: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2020:9
64: std::sys::pal::unix::thread::Thread::new::thread_start
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys/pal/unix/thread.rs:108:17
65: start_thread
66: clone"""
backtrace = """
0: 0x6021928eb3a3 - backtrace::backtrace::libunwind::trace::hf07e4a9b3540fe32
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.69/src/backtrace/libunwind.rs:93:5
backtrace::backtrace::trace_unsynchronized::h67e4196ea3f7ab74
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.69/src/backtrace/mod.rs:66:5
backtrace::backtrace::trace::hc7ba00e1051b4b01
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.69/src/backtrace/mod.rs:53:14
backtrace::capture::Backtrace::create::hcbf9c37c697d7aa9
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.69/src/capture.rs:176:9
backtrace::capture::Backtrace::new::h6ec2e445009cf64c
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.69/src/capture.rs:140:22
1: 0x60218eee690f - aptos_crash_handler::handle_panic::h5e65927cdb8e50da
at /aptos/crates/crash-handler/src/lib.rs:37:38
aptos_crash_handler::setup_panic_handler::{{closure}}::h995ae9b3bfa2c5d6
at /aptos/crates/crash-handler/src/lib.rs:29:9
2: 0x602192b6f500 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h022ca2c0d8c21c9e
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2034:9
std::panicking::rust_panic_with_hook::h0ad14d90dcf5224f
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:783:13
3: 0x602192b6f242 - std::panicking::begin_panic_handler::{{closure}}::h4a1838a06f542647
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:657:13
4: 0x602192b6dd36 - std::sys_common::backtrace::__rust_end_short_backtrace::h77cc4dc3567ca904
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:171:18
5: 0x602192b6ef74 - rust_begin_unwind
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5
6: 0x602192b9ca75 - core::panicking::panic_fmt::h940d4fd01a4b4fd1
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14
7: 0x602192b9cfd3 - core::result::unwrap_failed::h5119205a73b72b0d
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs:1654:5
8: 0x60218f45c9ce - core::result::Result<T,E>::unwrap::h5ab4d406fb3e6a0b
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs:1077:23
aptos_indexer_grpc_table_info::table_info_service::TableInfoService::get_highest_known_version::{{closure}}::ha30a3a6ee1b6c7c6
at /aptos/ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs:489:34
aptos_indexer_grpc_table_info::table_info_service::TableInfoService::run::{{closure}}::h1fb2c0b58271606d
at /aptos/ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs:89:67
aptos_indexer_grpc_table_info::runtime::bootstrap::{{closure}}::h7569d0d704b9430c
at /aptos/ecosystem/indexer-grpc/indexer-grpc-table-info/src/runtime.rs:109:22
9: 0x60218f44fc7b - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hd545ab7e70a28e79
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:331:17
tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h608e5b031c207f4b
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9
tokio::runtime::task::core::Core<T,S>::poll::h5d7e82741b6e7d96
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:320:30
10: 0x60218f4324b3 - tokio::runtime::task::harness::poll_future::{{closure}}::hc5e14b640432e8a3
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:500:19
<core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h8058176cb7755ab7
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
std::panicking::try::do_call::hb76601cb3917481d
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
std::panicking::try::h6ce564b1a6bf7f5c
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
std::panic::catch_unwind::h649c26f5b590f71e
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
tokio::runtime::task::harness::poll_future::hc79aa069c474028c
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:488:18
tokio::runtime::task::harness::Harness<T,S>::poll_inner::h245e31e3811972ab
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:209:27
tokio::runtime::task::harness::Harness<T,S>::poll::h60b85dcaa9089e14
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:154:15
11: 0x6021927c04f3 - tokio::runtime::task::raw::RawTask::poll::h939c040c00de17ca
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:201:18
tokio::runtime::task::LocalNotified<S>::run::h4da86429800bf6f5
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/mod.rs:436:9
tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h0469c7a903e1b0a2
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:598:18
tokio::runtime::coop::with_budget::h69827bda1d004d33
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
tokio::runtime::coop::budget::h05a2703f3385c108
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h19d777730477937a
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:597:9
12: 0x6021927bf5ea - tokio::runtime::scheduler::multi_thread::worker::Context::run::ha987d5c262f270e3
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:548:24
13: 0x6021927ab2a0 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h188dfdf770813901
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:513:21
tokio::runtime::context::scoped::Scoped<T>::set::hee3b4b536971cc0b
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/scoped.rs:40:9
tokio::runtime::context::set_scheduler::{{closure}}::h177c1db799e06512
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:26
std::thread::local::LocalKey<T>::try_with::hf09f5e96fc1c16a1
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:284:16
std::thread::local::LocalKey<T>::with::hc3b6f71a855c1582
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:260:9
tokio::runtime::context::set_scheduler::h38ed846e041210a1
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:17
14: 0x602192799c57 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h936a133b15bcfb77
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:508:9
tokio::runtime::context::runtime::enter_runtime::h85e5b9d5fe773a08
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
15: 0x6021927bed1b - tokio::runtime::scheduler::multi_thread::worker::run::h7c60b69e856e60ea
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:500:5
16: 0x60219279b21a - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h89c38ae8a9292e5c
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:466:45
<tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h3587c053437142fd
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/task.rs:42:21
17: 0x6021927ba9d1 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h30f605e2ff3d644d
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:331:17
tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h6b9db5cb13afaa28
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9
tokio::runtime::task::core::Core<T,S>::poll::hf920e48a7400be57
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:320:30
18: 0x60219278f2aa - tokio::runtime::task::harness::poll_future::{{closure}}::h4a5d235e31b3246f
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:500:19
<core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::ha412108464dcb0d2
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
std::panicking::try::do_call::h7deb1b85ba564761
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
std::panicking::try::h6b7188d56f9e5eb7
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
std::panic::catch_unwind::h7587d2a6d7c36da1
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
tokio::runtime::task::harness::poll_future::h5c854cf3fbd8df14
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:488:18
tokio::runtime::task::harness::Harness<T,S>::poll_inner::h8489bec210e087e9
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:209:27
tokio::runtime::task::harness::Harness<T,S>::poll::h6978499b31be20c2
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:154:15
tokio::runtime::task::raw::poll::h38847b94875ade96
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:271:5
19: 0x6021927a3d92 - tokio::runtime::task::raw::RawTask::poll::h939c040c00de17ca
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:201:18
tokio::runtime::task::UnownedTask<S>::run::h6d006efeebd476c5
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/mod.rs:473:9
tokio::runtime::blocking::pool::Task::run::h1fedfa477013357c
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:160:9
tokio::runtime::blocking::pool::Inner::run::h1bde900840449bec
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:518:17
20: 0x6021927a8d41 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h7e69608d817d98d6
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:476:13
std::sys_common::backtrace::__rust_begin_short_backtrace::h08be41586878d5cf
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
21: 0x6021927b84eb - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hb6e55083e018f9a4
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/mod.rs:528:17
<core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h90cdcb79e15c68eb
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
std::panicking::try::do_call::h1aa4807a92c19afb
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
std::panicking::try::h889729420105406a
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
std::panic::catch_unwind::h5b4b07131b374de3
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
std::thread::Builder::spawn_unchecked_::{{closure}}::hc18508185c075323
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/mod.rs:527:30
core::ops::function::FnOnce::call_once{{vtable.shim}}::h0f957443a4c1be98
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
22: 0x602192b747e5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h19b9e642d37e7272
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2020:9
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h97265befc434d3ae
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2020:9
std::sys::pal::unix::thread::Thread::new::thread_start::h420dad5cf01a9f35
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys/pal/unix/thread.rs:108:17
23: 0x7f5f20e58ea7 - start_thread
24: 0x7f5f20c2cacf - clone
25: 0x0 - <unknown>
"""
To reproduce
Folllow below steps to steadily reproduce.
- In a good enough machine, deploy a PFN using docker following docs. note that since i want to enable grpc indexer, i change fullnode.yaml into
base:
role: "full_node"
data_dir: "/opt/aptos/data"
waypoint:
from_file: "/opt/aptos/etc/waypoint.txt"
execution:
genesis_file_location: "/opt/aptos/etc/genesis.blob"
full_node_networks:
- network_id: "public"
discovery_method: "onchain"
listen_address: "/ip4/0.0.0.0/tcp/6182"
storage:
enable_indexer: true
indexer_grpc:
enabled: true
use_data_service_interface: true
address: 0.0.0.0:50051
processor_task_count: 10
processor_batch_size: 100
output_batch_size: 100
indexer_table_info:
table_info_service_mode: IndexingOnly
api:
enabled: true
address: "0.0.0.0:8080"
- when the fullnode starts syncing, i am able to tell fast sync progress using
curl 127.0.0.1:9101/metrics 2> /dev/null | grep "aptos_state_sync_version{.*\"synced_states\"}" | awk '{print $2}'following docs. i observe it goes all the way to at least 333911999 (my last observation) - when i then run the above curl command again shortly after step 2, i notice the aptos-fullnode container exited. checking docker logs give me above mentioned logs.
Expected Behavior
I expect the fullnode to sync to latest state and I should be able to call grpc endpoints such as grpcurl -max-msg-sz 20000000 -d '{ "starting_version": 1, "transactions_count": 1 }' -import-path protos/proto -proto aptos/indexer/v1/raw_data.proto -plaintext 127.0.0.1:50051 aptos.indexer.v1.RawData/GetTransactions
System information
Linux Ubuntu 22
