'icinga2', stop reason = signal SIGBUS

[lvasiliev]

We build icinga2 with debug info and are ready to provide additional information about this problem.

pid 47744 (icinga2), jid 2, uid 183: exited on signal 10 (core dumped)

Your Environment

FreeBSD 11.4-RELEASE-p9

root@icinga2-s:~ # icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.3-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Unknown
  Platform version: Unknown
  Kernel: FreeBSD
  Kernel version: 11.4-RELEASE-p9
  Architecture: amd64

Build information:
  Compiler: Clang 10.0.0
  Build host: icinga2-s
  OpenSSL version: OpenSSL 1.0.2u-freebsd  20 Dec 2019

Additional context

(lldb) target create "/usr/local/lib/icinga2/sbin/icinga2" --core "/var/coredumps/183.icinga2.47744.core"
Core file '/var/coredumps/183.icinga2.47744.core' (x86_64) was loaded.
(lldb) bt
* thread #1, name = 'icinga2', stop reason = signal SIGBUS
  * frame #0: 0x0000000000477804 icinga2`icinga::intrusive_ptr_add_ref(icinga::Object*) + 4
    frame #1: 0x00000000007aaf11 icinga2`icinga::Checkable::ProcessCheckResult(boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::MessageOrigin> const&) + 5665
    frame #2: 0x00000000007f53e9 icinga2`icinga::ClusterEvents::CheckResultAPIHandler(boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&) + 3033
    frame #3: 0x00000000006810b5 icinga2`std::__1::__function::__func<icinga::Value (*)(boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&), std::__1::allocator<icinga::Value (*)(boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&)>, icinga::Value (boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&)>::operator()(boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&) + 21
    frame #4: 0x0000000000646db2 icinga2`icinga::JsonRpcConnection::MessageHandler(icinga::String const&) + 1522
    frame #5: 0x00000000006462d7 icinga2`icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) + 519
    frame #6: 0x0000000000673d7f icinga2`boost::coroutines::detail::push_coroutine_object<boost::coroutines::pull_coroutine<void>, void, boost::asio::detail::coro_entry_point<boost::asio::executor_binder<void (*)(), boost::asio::io_context::strand>, void icinga::IoEngine::SpawnCoroutine<boost::asio::io_context::strand, icinga::JsonRpcConnection::Start()::$_70>(boost::asio::io_context::strand&, icinga::JsonRpcConnection::Start()::$_70)::'lambda'(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >)>&, boost::coroutines::basic_standard_stack_allocator<boost::coroutines::stack_traits> >::run() + 479
    frame #7: 0x0000000000673b94 icinga2`void boost::coroutines::detail::trampoline_push_void<boost::coroutines::detail::push_coroutine_object<boost::coroutines::pull_coroutine<void>, void, boost::asio::detail::coro_entry_point<boost::asio::executor_binder<void (*)(), boost::asio::io_context::strand>, void icinga::IoEngine::SpawnCoroutine<boost::asio::io_context::strand, icinga::JsonRpcConnection::Start()::$_70>(boost::asio::io_context::strand&, icinga::JsonRpcConnection::Start()::$_70)::'lambda'(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >)>&, boost::coroutines::basic_standard_stack_allocator<boost::coroutines::stack_traits> > >(boost::context::detail::transfer_t) + 84
    frame #8: 0x00000008014d2a8f libboost_context.so.1.72.0`make_fcontext + 47
(lldb) 

[Al2Klimov]

@bsdlme Please could you take a look at this?

[bsdlme]

If I knew anything about C++ I could be of her help here. Unfortunately I don't. :(

[Al2Klimov]

@sthen ? Anyone?

[davehayes]

This might be related to #7539

[Al2Klimov]

We build icinga2 with debug info

@lvasiliev But your LLDB shows no lines. :(

[Al2Klimov]

At least does it work on a more recent FreeBSD?

[sthen]

FWIW I haven't seen any icinga crashes on OpenBSD recently. As far as reports go, this is quite sparse, no details of config, no debug logs, minimal detail in the backtrack.. @lvasiliev you might want to try running "icinga2 daemon -x debug" under script(1) at least then you'll see what it's doing in the run-up to the crash.

[lvasiliev]

Hello!

Now we are currently using the following environment:

root@icinga2:~ # icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.2-1)
...
System information:
  Platform: FreeBSD
  Platform version: 12.3-RELEASE-p2
  Kernel: FreeBSD
  Kernel version: 12.3-RELEASE-p1
  Architecture: amd64

Build information:
  Compiler: Clang 10.0.1
  Build host: FreeBSD-12_2-RELEASE-amd64-icinga211210-icinga-job-01
  OpenSSL version: OpenSSL 1.1.1l-freebsd  24 Aug 2021
...

The number of errors has decreased, but most recently icinga2 process was killed 11 signal, but in this case we don't have coredump files.

/var/log/messages.5.bz2:Mar 11 18:38:48 icinga-master-2 kernel: pid 21749 (icinga2), jid 1, uid 183: exited on signal 11

Perhaps this bug can be closed or frozen. When we have time, we will revert the icinga2 build with debug info and turn on coredump. And we will try to collect more information.

We use a distributed configuration with 2 masters and satellites.