SIGSEGV after v5.0.0 upgrade, master branch commit id f00a05e7cf81e9422e9cf86b63b09ec26e312170

[Squirrelies]

Issue

Experiencing intermittent crashes after upgrading to v5.0.0. DEBUG build as of commit f00a05e7cf81e9422e9cf86b63b09ec26e312170

Built using a mix of instructions from How to Debug a Seg Fault | Trunk Recorder and various internet sources on applying ulimits to systemd services.

Willing to make and try new builds. Let me know if there is any additional info I can provide.

Systemd service file

squirrelies@sdr1:~/trunk-recorder$ cat /etc/systemd/system/trunk-recorder.service
[Unit]
Description=Trunk-Recorder
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=squirrelies
Group=squirrelies
LimitCORE=infinity
Restart=on-failure
WorkingDirectory=/home/squirrelies/tr-live
ExecStart=trunk-recorder --config=config.json

[Install]
WantedBy=multi-user.target

Core dumps

squirrelies@sdr1:~/trunk-recorder$ ls -la /var/lib/apport/coredump/
total 12064584
drwxr-xr-x 2 root        root       4096 Aug 19 13:44 .
drwxr-xr-x 3 root        root       4096 Apr 23 09:40 ..
-r-------- 1 squirrelies root 4119076864 Aug 18 21:49 core._usr_local_bin_trunk-recorder.1000.89a599dd-081d-4a9c-bb1e-37da59950c70.155881.14653388
-r-------- 1 squirrelies root 4118155264 Aug 19 05:01 core._usr_local_bin_trunk-recorder.1000.89a599dd-081d-4a9c-bb1e-37da59950c70.293521.35614011
-r-------- 1 squirrelies root 4116881408 Aug 19 13:45 core._usr_local_bin_trunk-recorder.1000.89a599dd-081d-4a9c-bb1e-37da59950c70.346624.38200911

Core dump stack trace

squirrelies@sdr1:~$ gdb trunk-recorder /var/lib/apport/coredump/core._usr_local_bin_trunk-recorder.1000.89a599dd-081d-4a9c-bb1e-37da59950c70.346624.38200911
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from trunk-recorder...

warning: Can't open file /SYSV00000000 (deleted) during file-backed mapping note processing

warning: Can't open file /home/squirrelies/orcexec.B6ld7T (deleted) during file-backed mapping note processing
[New LWP 386901]
[New LWP 346624]
[New LWP 346629]
[New LWP 346630]
[New LWP 386903]
[New LWP 386647]
[New LWP 346632]
[New LWP 346628]
[New LWP 346634]
[New LWP 346641]
[New LWP 346625]
[New LWP 346631]
[New LWP 346626]
[New LWP 346627]
[New LWP 346633]
[New LWP 386644]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `trunk-recorder --config=config.json'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count (__r=..., this=<optimized out>) at /usr/include/c++/13/bits/shared_ptr_base.h:1522

warning: Source file is more recent than executable.
1522          __shared_ptr(const __shared_ptr&) noexcept = default;
[Current thread is 1 (Thread 0x7348dd2006c0 (LWP 386901))]
(gdb) bt full
#0  std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count (__r=..., this=<optimized out>) at /usr/include/c++/13/bits/shared_ptr_base.h:1522
No locals.
#1  std::__shared_ptr<gr::buffer, (__gnu_cxx::_Lock_policy)2>::__shared_ptr (this=<optimized out>, this=<optimized out>) at /usr/include/c++/13/bits/shared_ptr_base.h:1522
No locals.
#2  std::shared_ptr<gr::buffer>::shared_ptr (this=<optimized out>, this=<optimized out>) at /usr/include/c++/13/bits/shared_ptr.h:204
No locals.
#3  gr::buffer_reader::buffer (this=<optimized out>, this=<optimized out>) at /usr/src/gnuradio-3.10.9.2-1.1ubuntu2/gnuradio-runtime/lib/../include/gnuradio/buffer_reader.h:82
No locals.
#4  gr::block_detail::post_work_cleanup (this=this@entry=0x5787203d71d0) at /usr/src/gnuradio-3.10.9.2-1.1ubuntu2/gnuradio-runtime/lib/../include/gnuradio/block_detail.h:208
        i = 0
#5  0x0000734b7fab8d0f in gr::block_executor::run_one_iteration (this=<optimized out>) at /usr/src/gnuradio-3.10.9.2-1.1ubuntu2/gnuradio-runtime/lib/block_executor.cc:665
        i = <optimized out>
        n = 3376
        msg = <optimized out>
        i = <optimized out>
        try_again = <optimized out>
        noutput_items = <optimized out>
        max_items_avail = <optimized out>
        max_noutput_items = <optimized out>
        new_alignment = <optimized out>
        alignment_state = <optimized out>
        output_idx = 0
        m = <optimized out>
        d = 0x5787203d71d0
#6  0x0000734b7fb02fa8 in gr::tpb_thread_body::tpb_thread_body (this=<optimized out>, block=..., start_sync=..., max_noutput_items=<optimized out>, this=<optimized out>, block=..., start_sync=..., max_noutput_items=<optimized out>)
    at /usr/src/gnuradio-3.10.9.2-1.1ubuntu2/gnuradio-runtime/lib/tpb_thread_body.cc:94
        s = <optimized out>
        msg = std::shared_ptr<pmt::pmt_base> (empty) = {get() = <optimized out>}
        max_nmsgs = 8192
        logger = {_name = "tpb_thread_body", d_logger = std::shared_ptr<spdlog::logger> (use count 1, weak count 0) = {get() = 0x734aa0002cc0}}
        d = 0x5787203d71d0
        p = <optimized out>
        d = <optimized out>
        s = <optimized out>
        msg = <optimized out>
        p = <optimized out>
        max_nmsgs = <optimized out>
        logger = <optimized out>
        i = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        guard = <optimized out>
        timeout = <optimized out>
        guard = <optimized out>
#7  0x0000734b7faf493b in gr::tpb_container::operator() (this=this@entry=0x5787203858e0) at /usr/src/gnuradio-3.10.9.2-1.1ubuntu2/gnuradio-runtime/lib/scheduler_tpb.cc:38
        body = {d_exec = {d_block = std::shared_ptr<gr::block> (use count 9, weak count 2) = {get() = 0x57871f746dc0}, d_logger = std::shared_ptr<gr::logger> (use count 1, weak count 0) = {get() = 0x734aa00011d0},
            d_debug_logger = std::shared_ptr<gr::logger> (use count 1, weak count 0) = {get() = 0x734a9c000f00}, d_ninput_items_required = std::vector of length 0, capacity 0, d_ninput_items = std::vector of length 0, capacity 0,
            d_input_items = std::vector of length 0, capacity 0, d_input_done = std::vector<bool> of length 0, capacity 0, d_output_items = std::vector of length 1, capacity 1 = {0x734b747b5970},
            d_start_nitems_read = std::vector of length 0, capacity 0, d_returned_tags = std::vector of length 0, capacity 0, d_max_noutput_items = 8192, d_use_pc = false}}
#8  0x0000734b7faf49e4 in gr::thread::thread_body_wrapper<gr::tpb_container>::operator() (this=0x5787203858e0) at /usr/src/gnuradio-3.10.9.2-1.1ubuntu2/gnuradio-runtime/lib/../include/gnuradio/thread/thread_body_wrapper.h:49
No locals.
#9  std::__invoke_impl<void, gr::thread::thread_body_wrapper<gr::tpb_container>&> (__f=...) at /usr/include/c++/13/bits/invoke.h:61
No locals.
#10 std::__invoke_r<void, gr::thread::thread_body_wrapper<gr::tpb_container>&> (__fn=...) at /usr/include/c++/13/bits/invoke.h:111
No locals.
#11 std::_Function_handler<void (), gr::thread::thread_body_wrapper<gr::tpb_container> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/13/bits/std_function.h:290
No locals.
#12 0x0000734b7fa3f97f in boost::(anonymous namespace)::thread_proxy (param=<optimized out>) at libs/thread/src/pthread/thread.cpp:179
        thread_info = <optimized out>
        lock = <optimized out>
#13 0x0000734b7ec9ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {126756784703168, -7891563696200645552, 126756784703168, -216, 2, 140725689142272, -7891563696246782896, -7891204790864214960}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {
              prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#14 0x0000734b7ed29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
No locals.

Partial log from sudo journalctl -u trunk-recorder.service --since "2024-08-19 13:44:50" --until "2024-08-19 13:45:06"

Aug 19 13:44:56 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:56.014967] (error)   [wiwinnouta] Retuning to Control Channel: 859.462500 MHz
Aug 19 13:44:56 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:56.015146] (info)            - System Source 0 - Min Freq: 858.075000 MHz Max Freq: 860.375000 MHz
Aug 19 13:44:56 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:56.015387] (error)   [wiwinnouta]         Control Channel Message Decode Rate: 1/sec, count:  3
Aug 19 13:44:56 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:56.157811] (error)   P25 Parse error, s:  Len: 0
Aug 19 13:44:57 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:57.193561] (error)   P25 Parse error, s:  Len: 0
Aug 19 13:44:58 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:58.240040] (error)   P25 Parse error, s:  Len: 0
Aug 19 13:44:59 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:59.012401] (error)   [wiwinnouta] Retuning to Control Channel: 858.462500 MHz
Aug 19 13:44:59 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:59.012582] (info)            - System Source 0 - Min Freq: 858.075000 MHz Max Freq: 860.375000 MHz
Aug 19 13:44:59 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:59.012800] (error)   [wiwinnouta]         Control Channel Message Decode Rate: 0/sec, count:  1
Aug 19 13:44:59 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:59.104097] (error)   [wiwinnouta] Retuning to Control Channel: 857.012500 MHz
Aug 19 13:44:59 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:59.104152] (info)            - System Source 1 - Min Freq: 855.821875 MHz Max Freq: 858.121875 MHz
Aug 19 13:44:59 sdr1 trunk-recorder[346624]: [2024-08-19 13:44:59.113530] (info)            Xlating Channelizer decimator - freq_xlating taps: 91 Decim: 4 Resampled Rate: 24000 Lowpass Taps: 83
Aug 19 13:45:05 sdr1 systemd[1]: trunk-recorder.service: Main process exited, code=dumped, status=11/SEGV
Aug 19 13:45:05 sdr1 systemd[1]: trunk-recorder.service: Failed with result 'core-dump'.
Aug 19 13:45:05 sdr1 systemd[1]: trunk-recorder.service: Consumed 8h 56min 36.054s CPU time.
Aug 19 13:45:05 sdr1 systemd[1]: trunk-recorder.service: Scheduled restart job, restart counter is at 3.
Aug 19 13:45:05 sdr1 systemd[1]: Started trunk-recorder.service - Trunk-Recorder.
Aug 19 13:45:05 sdr1 trunk-recorder[387707]: [2024-08-19 13:45:05.484693] (info)
Aug 19 13:45:05 sdr1 trunk-recorder[387707]: -------------------------------------
Aug 19 13:45:05 sdr1 trunk-recorder[387707]:      Trunk Recorder
Aug 19 13:45:05 sdr1 trunk-recorder[387707]: -------------------------------------
Aug 19 13:45:05 sdr1 trunk-recorder[387707]: [2024-08-19 13:45:05.484726] (info)
Aug 19 13:45:05 sdr1 trunk-recorder[387707]: -------------------------------------
Aug 19 13:45:05 sdr1 trunk-recorder[387707]: INSTANCE
Aug 19 13:45:05 sdr1 trunk-recorder[387707]: -------------------------------------
Aug 19 13:45:05 sdr1 trunk-recorder[387707]: [2024-08-19 13:45:05.484740] (info)   Using Config file: config.json
Aug 19 13:45:05 sdr1 trunk-recorder[387707]: [2024-08-19 13:45:05.484749] (info)   Trunk-Recorder: Version: 5.0.0