Assertion in ThreadLocalStorage::thread_local_node
I'm trying to get a full node working using mytonctrl and I'm having problems getting the validator up and running. I got the following stacktrace in /var/ton-work/log:
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f394492f850]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f394489dac3]
td::detail::ThreadPthread::run_thread
td::LambdaDestructor<td::detail::ThreadPthread::ThreadPthread<td::actor::core::Scheduler::start
td::LambdaDestructor<td::detail::ThreadPthread::ThreadPthread<td::actor::core::Scheduler::start
td::actor::core::CpuWorker::run
td::actor::core::ActorExecutor::start
td::actor::core::ActorExecutor::flush_one_signal
ton::validator::CellDb::start_up
vm::DynamicBagOfCellsDb::create
td::detail::process_check_error
td::Logger::~Logger
td::detail::TsFileLog::append
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f394484b520]
td::default_failure_signal_handler
td::Stacktrace::print_to_stderr
The faulty part was in vm::DynamicBagOfCellsDb::create. Somehow this assertion failed:
static constexpr int32 MAX_THREAD_ID = 128;
std::atomic<int32> max_thread_id_{MAX_THREAD_ID};
std::array<Node, MAX_THREAD_ID> nodes_;
Node &thread_local_node() {
auto thread_id = get_thread_id();
- CHECK(0 <= thread_id && static_cast<size_t>(thread_id) < nodes_.size());
return nodes_[thread_id];
}
The max thread id in ThreadLocalStorage.h is 128, but it is 256 in Scheduler.h, is this an intended behaviour?
More information: OS: Ubuntu 22.04.4 LTS x86_64 Kernel: 5.15.0-101-generic CPU: AMD EPYC 9654 (192) @ 2.400GHz Memory: 1031737MiB
Mytonctrl seems to clone and build the WIP version of the repo so it was dd5540d69e25f08a1c63760d3afb033208d9c99b but I'm not sure.
My server has 192 cores so mytonctrl uses --threads 191 for the validator-engine which causes the problem. I fixed it by changing the --threads argument in validator.service:
sudo systemctl stop validator
sudo nano /etc/systemd/system/validator.service
# lower the --threads number in nano editor
sudo systemctl daemon-reload
sudo systemctl start validator
I can open a PR in mytonctrl to fix it but I'm unsure which max thread id was intended, 128 or 256.