freeswitch FreeSwitch crashs every day

trafficstars

Hello,

FreeSwitch crashs one or two times every 24 hours , the FS version is 1.10.7 ,the OS is centos 7 ,the only error messages in /var/log/messages is : Apr 25 12:19:43 localhost systemd: freeswitch.service: main process exited, code=killed, status=8/FPE Apr 25 12:19:43 localhost systemd: Unit freeswitch.service entered failed state. Apr 25 12:19:43 localhost systemd: freeswitch.service failed.

Backtrace:

Core was generated by `/usr/local/fs1.10.7/bin/freeswitch -nonat'.
Program terminated with signal SIGFPE, Arithmetic exception.
#0  0x00007f1651eb87e8 in su_block_add () from /usr/local/lib/libsofia-sip-ua.so.0
[Current thread is 1 (LWP 45853)]
(gdb) where
#0  0x00007f36d8db1c3d in su_block_add (b=0x7f36945a3ce0, p=0x7f3694a73170) at su_alloc.c:339
#1  0x00007f36d8db21f2 in sub_alloc (home=0x7f36944eea20, sub=0x7f36945a3ce0, size=80, 
    zero=do_calloc) at su_alloc.c:527
#2  0x00007f36d8db3d2f in su_zalloc (home=0x7f36944eea20, size=80) at su_alloc.c:1574
#3  0x00007f36d8d3dcf9 in outgoing_make_a_aaaa_query (orq=0x7f36945a4a40) at nta.c:10343
#4  0x00007f36d8d3d32e in outgoing_resolve_next (orq=0x7f36945a4a40) at nta.c:10124
#5  0x00007f36d8d3eda7 in outgoing_answer_srv (orq=0x7f36945a4a40, q=0x7f36945a4c20, 
    answers=0x7f369411dfa0) at nta.c:10707
#6  0x00007f36d8da5fd9 in sres_query_report_error (q=0x7f36945a4c20, answers=0x7f369411dfa0)
    at sres.c:2990
#7  0x00007f36d8da6368 in sres_resend_dns_query (res=0x7f3694002960, q=0x7f36945a4c20, 
    timeout=0) at sres.c:3090
#8  0x00007f36d8da725b in sres_resolver_report_error (res=0x7f3694002960, socket=129, 
    errcode=111, remote=0x7f36ab318450, remotelen=16, 
    info=0x7f36ab3183d0 "icmp type=3 code=3 reported by 127.0.0.1") at sres.c:3439
#9  0x00007f36d8da6f29 in sres_resolver_error (res=0x7f3694002960, socket=129) at sres.c:3353
#10 0x00007f36d8db078d in sres_sofia_poll (magic=0x7f36ac001180, w=0x7f3694000ff4, 
    reg=0x7f3694003790) at sresolv.c:357
#11 0x00007f36d8db69ac in su_epoll_port_wait_events (self=0x7f36940008c0, tout=0)
    at su_epoll_port.c:510
#12 0x00007f36d8dbc17f in su_port_wait_events (self=0x7f36940008c0, timeout=0) at su_port.h:318
#13 0x00007f36d8dbd481 in su_root_yield (self=0x7f3694001130) at su_root.c:904
#14 0x00007f36d8d71ffd in nua_stack_timer (nua=0x7f36ac001180, t=0x7f36940011e0, a=0x0)
    at nua_stack.c:739
#15 0x00007f36d8dc69f9 in su_timer_expire (timers=0x7f3694000908, timeout=0x7f36ab318cd8, 
    now=...) at su_timer.c:585
#16 0x00007f36d8db4ee2 in su_base_port_run (self=0x7f36940008c0) at su_base_port.c:339
#17 0x00007f36d8dbc1aa in su_port_run (self=0x7f36940008c0) at su_port.h:326
#18 0x00007f36d8dbd284 in su_root_run (self=0x7f3694001130) at su_root.c:819
#19 0x00007f36d8dbb933 in su_pthread_port_clone_main (varg=0x7f36d8abb4a0)
    at su_pthread_port.c:343
#20 0x00007f36e6ec9e65 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f36e651e88d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#22 0x0000000000000000 in ?? ()

Thanks!

Apr 29 '22 07:04 dxhgq-github

Please provide a proper gdb log. See Creating A Backtrace With gdb section: https://freeswitch.org/confluence/display/FREESWITCH/Debugging

Apr 29 '22 07:04 andywolk

What version of the sofia-sip library are you using?

Apr 29 '22 07:04 andywolk

sofia-sip 1.13.7

Apr 29 '22 07:04 dxhgq-github

Thread 146 (LWP 63539): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 145 (LWP 45867): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 144 (LWP 64468): #0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6 #1 0x00007f16542a3607 in rtp_common_read (rtp_session=<error reading variable: Cannot access memory at address 0xfffffe08>, payload_type=<error reading variable: Cannot access memory at address 0xfffffe00>, pmapP=<error reading variable: Cannot access memory at address 0xfffffdf8>, flags=<error reading variable: Cannot access memory at address 0xfffffdf0>, io_flags=<error reading variable: Cannot access memory at address 0xfffffdec>) at src/switch_rtp.c:7571 Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 143 (LWP 45890): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 142 (LWP 45893): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 141 (LWP 65096): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x000000000007a120 in ?? () #2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so #3 0x0000000002a1be40 in ?? () #4 0x00007f1493e53ca0 in ?? () #5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 140 (LWP 45895): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 139 (LWP 45905): #0 0x00007f165050a80d in sleep () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 138 (LWP 45876): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x00000000000f4240 in ?? () #2 0x0000000000000000 in ?? ()

Thread 137 (LWP 45879): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 136 (LWP 45880): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 135 (LWP 45883): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 134 (LWP 64687): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 133 (LWP 45886): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 132 (LWP 63929): #0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1654c38f48, mutex=0x7f1654c38ef8, timeout=5000000) at locks/unix/thread_cond.c:89 #2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493fc1d38, timeout=5000000) at misc/apr_queue.c:339 #3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493fc1d38, timeout=5000000) at src/switch_apr.c:1225 #4 0x00007f16541e8709 in switch_core_session_thread_pool_worker (thread=0x7f1628382dc0, obj=0x7f1628382c50) at src/switch_core_session.c:1792 #5 0x00007f165454f770 in dummy_worker (opaque=0x7f1628382dc0) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 131 (LWP 64389): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f16180a1210, mutex=0x7f16180a11c0) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f16180a13d0, obj=0x7f16180af708) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f16180a13d0) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 130 (LWP 64075): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 129 (LWP 64456): #0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 128 (LWP 64839): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x000000000007a120 in ?? () #2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so #3 0x0000000002a1be40 in ?? () #4 0x00007f1492dacca0 in ?? () #5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 127 (LWP 64774): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f15ac05ccc8, mutex=0x7f15ac05cc78) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f15ac05ce88, obj=0x7f15ac032d78) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f15ac05ce88) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 126 (LWP 64469): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f149c02aff0, mutex=0x7f149c02afa0) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f149c02b1b0, obj=0x7f149cb4cbe8) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f149c02b1b0) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 125 (LWP 65053): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 124 (LWP 65073): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 123 (LWP 65095): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 122 (LWP 65104): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x000000000007a120 in ?? () #2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so #3 0x0000000002a1be40 in ?? () #4 0x00007f155c145ca0 in ?? () #5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 121 (LWP 45898): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 120 (LWP 64225): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f150c041948, mutex=0x7f150c0418f8) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f150c041b08, obj=0x7f150c0d0098) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f150c041b08) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 119 (LWP 64452): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1498ab2a80, mutex=0x7f1498ab2a30) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f1498ab2c40, obj=0x7f1498730808) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f1498ab2c40) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 118 (LWP 64451): #0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6 #1 0x00007f16542a3607 in rtp_common_read (rtp_session=<error reading variable: Cannot access memory at address 0xfffffe08>, payload_type=<error reading variable: Cannot access memory at address 0xfffffe00>, pmapP=<error reading variable: Cannot access memory at address 0xfffffdf8>, flags=<error reading variable: Cannot access memory at address 0xfffffdf0>, io_flags=<error reading variable: Cannot access memory at address 0xfffffdec>) at src/switch_rtp.c:7571 Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 117 (LWP 65040): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 116 (LWP 65059): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 115 (LWP 45877): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x2af4f58, mutex=0x2af4f08) at src/switch_apr.c:371 #2 0x00007f15ccec65cd in timer_thread_run (thread=0x2af5118, obj=0x0) at mod_spandsp_fax.c:223 #3 0x00007f165454f770 in dummy_worker (opaque=0x2af5118) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 114 (LWP 45882): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 113 (LWP 65214): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 112 (LWP 45884): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 111 (LWP 45885): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 110 (LWP 65071): #0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1654c38f48, mutex=0x7f1654c38ef8, timeout=5000000) at locks/unix/thread_cond.c:89 #2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493123d38, timeout=5000000) at misc/apr_queue.c:339 #3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493123d38, timeout=5000000) at src/switch_apr.c:1225 #4 0x00007f16541e8709 in switch_core_session_thread_pool_worker (thread=0x7f14f002fb50, obj=0x7f14f002f9e0) at src/switch_core_session.c:1792 #5 0x00007f165454f770 in dummy_worker (opaque=0x7f14f002fb50) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 109 (LWP 65077): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14f893b140, mutex=0x7f14f893b0f0) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14f893b300, obj=0x7f14f806ec08) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f14f893b300) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 108 (LWP 45887): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 107 (LWP 45888): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 106 (LWP 45889): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 105 (LWP 45891): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 104 (LWP 45892): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 103 (LWP 45894): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 102 (LWP 45896): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 101 (LWP 45897): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so #2 0x00007f158e932b15 in rocketmq::TcpRemotingClient::boost_asio_work() () from /lib64/librocketmq.so #3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 100 (LWP 65207): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 99 (LWP 65166): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 98 (LWP 65060): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x000000000007a120 in ?? () #2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so #3 0x0000000002a1be40 in ?? () #4 0x00007f1529095ca0 in ?? () #5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 97 (LWP 45944): #0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6 #1 0x00007f1648f81b49 in read_packet (listener=<error reading variable: Cannot access memory at address 0x15c08d818>, event=<error reading variable: Cannot access memory at address 0x15c08d810>, timeout=<error reading variable: Cannot access memory at address 0x15c08d80c>) at mod_event_socket.c:1496 Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 96 (LWP 45986): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68 #2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f164c6008e8, data=0x7f15296abcf0) at misc/apr_queue.c:276 #3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f164c6008e8, data=0x7f15296abcf0) at src/switch_apr.c:1220 #4 0x00007f1654288b9e in switch_event_channel_deliver_thread (thread=0x7f1640010b10, obj=0x7f164c6008e8) at src/switch_event.c:3001 #5 0x00007f16541e8786 in switch_core_session_thread_pool_worker (thread=0x7f1640010b10, obj=0x7f16400109a0) at src/switch_core_session.c:1800 #6 0x00007f165454f770 in dummy_worker (opaque=0x7f1640010b10) at threadproc/unix/thread.c:151 #7 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #9 0x0000000000000000 in ?? ()

Thread 95 (LWP 61545): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 94 (LWP 63430): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 93 (LWP 63542): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 92 (LWP 63867): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 91 (LWP 64379): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 90 (LWP 64450): #0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 89 (LWP 63971): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 88 (LWP 63972): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 87 (LWP 64092): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f15bc4bf400, mutex=0x7f15bc4bf3b0) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f15bc4bf5c0, obj=0x7f15bc196e08) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f15bc4bf5c0) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 86 (LWP 65167): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x000000000007a120 in ?? () #2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so #3 0x0000000002a1be40 in ?? () #4 0x00007f1493282ca0 in ?? () #5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 85 (LWP 65151): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x000000000007a120 in ?? () #2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so #3 0x0000000002a1be40 in ?? () #4 0x00007f14935bcca0 in ?? () #5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 84 (LWP 65212): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 83 (LWP 64561): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 82 (LWP 64700): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 81 (LWP 64703): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 80 (LWP 64838): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 79 (LWP 64855): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 78 (LWP 64856): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x000000000007a120 in ?? () #2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so #3 0x0000000002dcc570 in ?? () #4 0x00007f14939f7ca0 in ?? () #5 0x00007f165431193c in switch_sleep (t=48022896) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 77 (LWP 64928): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 76 (LWP 64945): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 75 (LWP 65048): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 74 (LWP 65062): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 73 (LWP 65066): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 72 (LWP 65072): #0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 71 (LWP 65063): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x000000000007a120 in ?? () #2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so #3 0x0000000002a1be40 in ?? () #4 0x00007f1528f3fca0 in ?? () #5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 70 (LWP 65213): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f157000c780, mutex=0x7f157000c730) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f157000c940, obj=0x7f1570a7f758) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f157000c940) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 69 (Thread 0x7f1654c9e8c0 (LWP 45827)): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x00000000000f4240 in ?? () #2 0x00007fff556c0910 in ?? () #3 0x0000000002df4ca8 in ?? () #4 0x00007fff556c0910 in ?? () #5 0x00007f165431193c in switch_sleep (t=48188584) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 68 (LWP 45836): #0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6 #1 0x0000000000000005 in ?? () #2 0xffffffffffffffff in ?? () #3 0x0000000000000000 in ?? ()

Thread 67 (LWP 65087): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 66 (LWP 65101): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 65 (LWP 65103): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 64 (LWP 65108): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 63 (LWP 65113): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 62 (LWP 65161): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14d0004300, mutex=0x7f14d00042b0) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14d00044c0, obj=0x7f14d000ca28) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f14d00044c0) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 61 (LWP 65208): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x000000000007a120 in ?? () #2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so #3 0x0000000002a1be40 in ?? () #4 0x00007f1529b42ca0 in ?? () #5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 60 (LWP 65150): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000003473bc0 in ?? () #2 0x0000000000000000 in ?? ()

Thread 59 (LWP 65160): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 58 (LWP 65162): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 57 (LWP 45830): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x00000000000f4240 in ?? () #2 0x000000000003d000 in ?? () #3 0x00007f1654c38c40 in ?? () #4 0x00007f1654bd5d20 in ?? () #5 0x00007f165431193c in switch_sleep (t=139733888109632) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 56 (LWP 45832): #0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 55 (LWP 65215): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 54 (LWP 65216): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f15840402e0, mutex=0x7f1584040290) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f15840404a0, obj=0x7f1584b08c28) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f15840404a0) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 53 (LWP 65217): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 52 (LWP 45833): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68 #2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f164c7dbec8, data=0x7f164c53bd50) at misc/apr_queue.c:276 #3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f164c7dbec8, data=0x7f164c53bd50) at src/switch_apr.c:1220 #4 0x00007f16543029b3 in log_thread (t=0x7f164c6005a8, obj=0x0) at src/switch_log.c:501 #5 0x00007f165454f770 in dummy_worker (opaque=0x7f164c6005a8) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 51 (LWP 45851): #0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89 #2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf08d18, timeout=100000) at misc/apr_queue.c:339 #3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf08d18, timeout=100000) at src/switch_apr.c:1225 #4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x7f1648295b40, obj=0x29b4ff0) at sofia.c:3010 #5 0x00007f165454f770 in dummy_worker (opaque=0x7f1648295b40) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 50 (LWP 45870): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 49 (LWP 64464): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14a4020d78, mutex=0x7f14a4020d28) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14a4020f38, obj=0x7f14a4132318) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f14a4020f38) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 48 (LWP 45834): #0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x2733118, mutex=0x27330c8, timeout=500000) at locks/unix/thread_cond.c:89 #2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x2733088, data=0x7f164c314d58, timeout=500000) at misc/apr_queue.c:339 #3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x2733088, data=0x7f164c314d58, timeout=500000) at src/switch_apr.c:1225 #4 0x00007f16542690a9 in switch_scheduler_task_thread (thread=0x7f164c4fe5d8, obj=0x0) at src/switch_scheduler.c:191 #5 0x00007f165454f770 in dummy_worker (opaque=0x7f164c4fe5d8) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 47 (LWP 45850): #0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89 #2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf45d18, timeout=100000) at misc/apr_queue.c:339 #3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf45d18, timeout=100000) at src/switch_apr.c:1225 #4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x7f1648232b40, obj=0x29c2170) at sofia.c:3010 #5 0x00007f165454f770 in dummy_worker (opaque=0x7f1648232b40) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 46 (LWP 64308): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 45 (LWP 45852): #0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89 #2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cecbd18, timeout=100000) at misc/apr_queue.c:339 #3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cecbd18, timeout=100000) at src/switch_apr.c:1225 #4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x29be5f8, obj=0x29bb7d0) at sofia.c:3010 #5 0x00007f165454f770 in dummy_worker (opaque=0x29be5f8) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 44 (LWP 45878): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 43 (LWP 45881): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 42 (LWP 45932): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68 #2 0x00007f1654540ce1 in apr_queue_pop (queue=0x2dd3720, data=0x7f155c1bfd50) at misc/apr_queue.c:276 #3 0x00007f16541a8c37 in switch_queue_pop (queue=0x2dd3720, data=0x7f155c1bfd50) at src/switch_apr.c:1220 #4 0x00007f165426e3b4 in chat_thread_run (thread=0x2de3250, obj=0x2dd3720) at src/switch_loadable_module.c:911 #5 0x00007f165454f770 in dummy_worker (opaque=0x2de3250) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 41 (LWP 45933): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68 #2 0x00007f1654540ce1 in apr_queue_pop (queue=0x2de3280, data=0x7f155c182d50) at misc/apr_queue.c:276 #3 0x00007f16541a8c37 in switch_queue_pop (queue=0x2de3280, data=0x7f155c182d50) at src/switch_apr.c:1220 #4 0x00007f165426e3b4 in chat_thread_run (thread=0x2def5e0, obj=0x2de3280) at src/switch_loadable_module.c:911 #5 0x00007f165454f770 in dummy_worker (opaque=0x2def5e0) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 40 (LWP 45938): #0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 39 (LWP 45928): #0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 38 (LWP 45859): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68 #2 0x00007f1654540ce1 in apr_queue_pop (queue=0x29d5558, data=0x7f161fd7bd10) at misc/apr_queue.c:276 #3 0x00007f161fdb0ac2 in apt_consumer_task_run () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_unimrcp.so #4 0x00007f161fdb41a7 in apt_task_run (thread_handle=0x29ec528, data=0x29d93e8) at src/apt_task.c:591 #5 0x00007f165454f770 in dummy_worker (opaque=0x29ec528) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 37 (LWP 45863): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 36 (LWP 64592): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14e0029ab8, mutex=0x7f14e0029a68) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14e0029c78, obj=0x7f14e00eed18) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f14e0029c78) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 35 (LWP 45860): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 34 (LWP 64949): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 33 (LWP 45856): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68 #2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f1648c50ad8, data=0x7f162c426d48) at misc/apr_queue.c:276 #3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f1648c50ad8, data=0x7f162c426d48) at src/switch_apr.c:1220 #4 0x00007f1648cecd58 in sofia_msg_thread_run (thread=0x29b40d0, obj=0x7f1648c50ad8) at sofia.c:2295 #5 0x00007f165454f770 in dummy_worker (opaque=0x29b40d0) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 32 (LWP 45855): #0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89 #2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162c463d18, timeout=100000) at misc/apr_queue.c:339 #3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162c463d18, timeout=100000) at src/switch_apr.c:1225 #4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x7f162c68db50, obj=0x29c8950) at sofia.c:3010 #5 0x00007f165454f770 in dummy_worker (opaque=0x7f162c68db50) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 31 (LWP 65067): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x000000000007a120 in ?? () #2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so #3 0x0000000002a1be40 in ?? () #4 0x00007f14936b0ca0 in ?? () #5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644 Backtrace stopped: frame did not save the PC

Thread 30 (LWP 65076): #0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6 #1 0x00007f16542a3607 in rtp_common_read (rtp_session=<error reading variable: Cannot access memory at address 0xfffffe08>, payload_type=<error reading variable: Cannot access memory at address 0xfffffe00>, pmapP=<error reading variable: Cannot access memory at address 0xfffffdf8>, flags=<error reading variable: Cannot access memory at address 0xfffffdf0>, io_flags=<error reading variable: Cannot access memory at address 0xfffffdec>) at src/switch_rtp.c:7571 Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 29 (LWP 45837): #0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 28 (LWP 45849): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1628028ce8, mutex=0x7f1628028bf8) at src/switch_apr.c:371 #2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162d0846c0, obj=0x7f1628028b30) at src/switch_core_sqldb.c:2296 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f162d0846c0) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 27 (LWP 45854): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1624002598, mutex=0x7f16240024a8) at src/switch_apr.c:371 #2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162c5656c0, obj=0x7f16240023e0) at src/switch_core_sqldb.c:2296 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f162c5656c0) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 26 (LWP 45929): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x29f1548, mutex=0x29f14a8) at src/switch_apr.c:371 #2 0x00007f161c1698f4 in mod_verto_runtime () at mod_verto.c:6907 #3 0x00007f165426ac27 in switch_loadable_module_exec (thread=0x273ca38, obj=0x273c618) at src/switch_loadable_module.c:123 #4 0x00007f165454f770 in dummy_worker (opaque=0x273ca38) at threadproc/unix/thread.c:151 #5 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? ()

Thread 25 (LWP 45865): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 24 (LWP 45846): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 23 (LWP 45838): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x00000000000f4240 in ?? () #2 0x00007f164ad7dd30 in ?? () #3 0x00007f16541e1f7d in switch_core_expire_registration (force=0) at src/switch_core_sqldb.c:3557 #4 0x00007f165431193c in switch_sleep (t=1000000) at src/switch_time.c:644 #5 0x00007f16541db054 in switch_core_sql_db_thread (thread=0x7f164c600750, obj=0x0) at src/switch_core_sqldb.c:1541 #6 0x00007f165454f770 in dummy_worker (opaque=0x7f164c600750) at threadproc/unix/thread.c:151 #7 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #9 0x0000000000000000 in ?? ()

Thread 22 (LWP 45930): #0 0x00007f1650ef599d in fcntl () from /lib64/libpthread.so.0 #1 0x00007f15180029a0 in ?? () #2 0x0000000000002687 in ?? () #3 0x0000000000000000 in ?? ()

Thread 21 (LWP 45840): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 20 (LWP 45841): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 19 (LWP 45931): #0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0 #1 0x0000000000000000 in ?? ()

Thread 18 (LWP 45842): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 17 (LWP 45927): #0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 16 (LWP 45845): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 15 (LWP 45844): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 14 (LWP 45847): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1638028d08, mutex=0x7f1638028c18) at src/switch_apr.c:371 #2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162d20e6c0, obj=0x7f1638028b40) at src/switch_core_sqldb.c:2296 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f162d20e6c0) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 13 (LWP 45873): #0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f15a0fa7939 in ks_cond_timedwait (cond=0x2a1fb58, ms=10000) at /home/freeswitch/tmp/libks/src/ks_mutex.c:563 #2 0x00007f15cf1a7c4e in __manager_loop () at /home/freeswitch/tmp/20220405/signalwire-c/src/handle_manager.c:255 #3 0x00007f15cf1a7d0f in __manager_thread_wrapper (thread=0x2a1fcc8, data=0x0) at /home/freeswitch/tmp/20220405/signalwire-c/src/handle_manager.c:275 #4 0x00007f15a0fb5036 in thread_launch (args=0x2a1fcc8) at /home/freeswitch/tmp/libks/src/ks_thread.c:248 #5 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? ()

Thread 12 (LWP 45872): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 11 (LWP 45843): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 10 (LWP 65206): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14a812e0f0, mutex=0x7f14a812e0a0) at src/switch_apr.c:371 #2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14a812e2b0, obj=0x7f14a80a8bc8) at src/switch_ivr_async.c:1329 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f14a812e2b0) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 9 (LWP 45831): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68 #2 0x00007f1654540ce1 in apr_queue_pop (queue=0x24a1d98, data=0x7f164c751d48) at misc/apr_queue.c:276 #3 0x00007f16541a8c37 in switch_queue_pop (queue=0x24a1d98, data=0x7f164c751d48) at src/switch_apr.c:1220 #4 0x00007f16542819cd in switch_event_dispatch_thread (thread=0x7f164c7dbc40, obj=0x24a1d98) at src/switch_event.c:331 #5 0x00007f165454f770 in dummy_worker (opaque=0x7f164c7dbc40) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 8 (LWP 45848): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f16200280a8, mutex=0x7f1620027fb8) at src/switch_apr.c:371 #2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162d1496c0, obj=0x7f1620027ee0) at src/switch_core_sqldb.c:2296 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f162d1496c0) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 7 (LWP 45861): #0 0x00007f165053a953 in setegid () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 6 (LWP 45874): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68 #2 0x00007f1654540ce1 in apr_queue_pop (queue=0x29a4628, data=0x7f162c113d50) at misc/apr_queue.c:276 #3 0x00007f16541a8c37 in switch_queue_pop (queue=0x29a4628, data=0x7f162c113d50) at src/switch_apr.c:1220 #4 0x00007f1648d308fa in sofia_presence_event_thread_run (thread=0x29a54e0, obj=0x0) at sofia_presence.c:1625 #5 0x00007f165454f770 in dummy_worker (opaque=0x29a54e0) at threadproc/unix/thread.c:151 #6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #8 0x0000000000000000 in ?? ()

Thread 5 (LWP 45862): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 4 (LWP 45900): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68 #2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f153001b958, data=0x7f15c406d468) at misc/apr_queue.c:276 #3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f153001b958, data=0x7f15c406d468) at src/switch_apr.c:1220 #4 0x00007f165433b0ee in EventConsumer::pop (this=, block=block@entry=1, timeout=) at src/switch_cpp.cpp:110 #5 0x00007f155c1ebaf5 in _wrap_EventConsumer_pop (L=0x7f15300008c8) at mod_lua_wrap.cpp:5547 #6 0x00007f155c1f48a0 in luaD_precall () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so #7 0x00007f155c1ff29f in luaV_execute () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so #8 0x00007f155c1f4c9c in luaD_call () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so #9 0x00007f155c1ff0ac in luaV_execute () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so #10 0x00007f155c1f4c9c in luaD_call () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so #11 0x00007f155c1f425f in luaD_rawrunprotected () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so #12 0x00007f155c1f4f0d in luaD_pcall () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so #13 0x00007f155c1f275e in lua_pcallk () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so #14 0x00007f155c1d1b2d in docall (L=L@entry=0x7f15300008c8, narg=narg@entry=0, nresults=nresults@entry=0, perror=perror@entry=0, fatal=fatal@entry=1) at mod_lua.cpp:92 #15 0x00007f155c1d213f in lua_parse_and_execute (L=L@entry=0x7f15300008c8, input_code=, session=session@entry=0x0) at mod_lua.cpp:194 #16 0x00007f155c1d21d7 in lua_thread_run (thread=, obj=0x2c06050) at mod_lua.cpp:221 #17 0x00007f165454f770 in dummy_worker (opaque=0x2c061d8) at threadproc/unix/thread.c:151 #18 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #20 0x0000000000000000 in ?? ()

Thread 3 (LWP 45875): #0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x2a8d800, mutex=0x2a8d710) at src/switch_apr.c:371 #2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f161c1496c0, obj=0x2a8d660) at src/switch_core_sqldb.c:2296 #3 0x00007f165454f770 in dummy_worker (opaque=0x7f161c1496c0) at threadproc/unix/thread.c:151 #4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()

Thread 2 (LWP 45864): #0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6 #1 0x0000000000000000 in ?? ()

Thread 1 (LWP 45853): #0 0x00007f1651eb87e8 in su_block_add () from /usr/local/lib/libsofia-sip-ua.so.0 #1 0x00007f1651eb8d9d in sub_alloc () from /usr/local/lib/libsofia-sip-ua.so.0 #2 0x00007f1651eba8b1 in su_zalloc () from /usr/local/lib/libsofia-sip-ua.so.0 #3 0x00007f1651e42d4b in outgoing_make_a_aaaa_query () from /usr/local/lib/libsofia-sip-ua.so.0 #4 0x00007f1651e42350 in outgoing_resolve_next () from /usr/local/lib/libsofia-sip-ua.so.0 #5 0x00007f1651e43e88 in outgoing_answer_srv () from /usr/local/lib/libsofia-sip-ua.so.0 #6 0x00007f1651ead296 in sres_query_report_error () from /usr/local/lib/libsofia-sip-ua.so.0 #7 0x00007f1651ead643 in sres_resend_dns_query () from /usr/local/lib/libsofia-sip-ua.so.0 #8 0x00007f1651eae590 in sres_resolver_report_error () from /usr/local/lib/libsofia-sip-ua.so.0 #9 0x00007f1651eae24f in sres_resolver_error () from /usr/local/lib/libsofia-sip-ua.so.0 #10 0x00007f1651eb3eee in sres_sofia_poll () from /usr/local/lib/libsofia-sip-ua.so.0 #11 0x00007f1651ec5768 in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0 #12 0x00007f1651ec1b68 in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0 #13 0x00007f1651ebde18 in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0 #14 0x00007f1651ebeef2 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0 #15 0x00007f1651ec2924 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0 #16 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #18 0x0000000000000000 in ?? ()

Apr 29 '22 07:04 dxhgq-github

You need to install sofia sip debugging symbols. Also I don't think your sofia is the latest. And please attach the log, do not paste it.

Apr 29 '22 07:04 andywolk

Sounds like it is not a bug but some memory issues on your box. Information you provided does not show me that a crash happened. Line numbers do not match sofia-sip version you mentioned.

Apr 29 '22 07:04 andywolk

Ok, I will take a check at sofia-sip version , Thanks!

Apr 30 '22 02:04 dxhgq-github

Hello I have downloaded and rebuilt the latest version 1.13.7 of sofia-sip , after running 4 days , the FreeSwitch still crashs. error message is : Core was generated by `/usr/local/fs1.10.7/bin/freeswitch -nonat'. Program terminated with signal SIGFPE, Arithmetic exception. #0 0x00007f2c9e58381a in su_block_add (b=0x7f2c542282f0, p=0x7f2c542547f0) at su_alloc.c:342 342 h = (size_t)((uintptr_t)p % b->sub_n); [Current thread is 1 (LWP 92647)] (gdb) (gdb) bt #0 0x00007f2c9e58381a in su_block_add (b=0x7f2c542282f0, p=0x7f2c542547f0) at su_alloc.c:342 #1 0x00007f2c9e583dcf in sub_alloc (home=0x7f2c54250ef0, sub=0x7f2c542282f0, size=80, zero=do_calloc) at su_alloc.c:530 #2 0x00007f2c9e5858e3 in su_zalloc (home=0x7f2c54250ef0, size=80) at su_alloc.c:1577 #3 0x00007f2c9e50dd4b in outgoing_make_a_aaaa_query (orq=0x7f2c54254690) at nta.c:10463 #4 0x00007f2c9e50d350 in outgoing_resolve_next (orq=0x7f2c54254690) at nta.c:10244 #5 0x00007f2c9e50ee88 in outgoing_answer_srv (orq=0x7f2c54254690, q=0x7f2c5410b2c0, answers=0x7f2c541167b0) at nta.c:10830 #6 0x00007f2c9e5782c8 in sres_query_report_error (q=0x7f2c5410b2c0, answers=0x7f2c541167b0) at sres.c:2991 #7 0x00007f2c9e578675 in sres_resend_dns_query (res=0x7f2c54002990, q=0x7f2c5410b2c0, timeout=0) at sres.c:3091 #8 0x00007f2c9e5795c2 in sres_resolver_report_error (res=0x7f2c54002990, socket=590, errcode=111, remote=0x7f2c7d58c560, remotelen=16, info=0x7f2c7d58c4e0 "icmp type=3 code=3 reported by 127.0.0.1") at sres.c:3440 #9 0x00007f2c9e579281 in sres_resolver_error (res=0x7f2c54002990, socket=590) at sres.c:3354 #10 0x00007f2c9e57ef20 in sres_sofia_poll (magic=0x7f2c68001180, w=0x7f2c54001084, reg=0x7f2c540037c0) at sresolv.c:357 #11 0x00007f2c9e59079a in su_epoll_port_wait_events (self=0x7f2c540008c0, tout=690) at su_epoll_port.c:510 #12 0x00007f2c9e58cb9a in su_base_port_run (self=0x7f2c540008c0) at su_base_port.c:349 #13 0x00007f2c9e588e4a in su_port_run (self=0x7f2c540008c0) at su_port.h:326 #14 0x00007f2c9e589f24 in su_root_run (self=0x7f2c54001130) at su_root.c:819 #15 0x00007f2c9e58d956 in su_pthread_port_clone_main (varg=0x7f2c91233460) at su_pthread_port.c:343 #16 0x00007f2c9d5b9e65 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f2c9cc0e88d in __libc_ifunc_impl_list () from /lib64/libc.so.6 #18 0x0000000000000000 in ?? () (gdb)

backtrace1.log

May 05 '22 02:05 dxhgq-github

Please help , thanks

May 06 '22 08:05 dxhgq-github

Please wrap backtraces into a code block when you comment.

Jun 16 '22 10:06 andywolk

Based on your last backtrace please print some debugging information:

frame 2
p *home

Jun 16 '22 10:06 andywolk

thank you for replay !

Jun 17 '22 07:06 dxhgq-github

(gdb) (gdb) (gdb) frame 2 #2 0x00007ff892b5992d in su_zalloc (home=0x7ff86028d060, size=80) at su_alloc.c:1577 1577 data = sub_alloc(home, MEMLOCK(home), size, (enum sub_zero)1); (gdb) p *home $1 = {suh_size = 1620230464, suh_blocks = 0x7ff8613c3900, suh_lock = 0x0} (gdb)

thank you !

Jun 17 '22 07:06 dxhgq-github

whether it is a IPv6 resolve problem ? but I do not configure any call flow about local resolve, and ping 127.0.0.1 is alive , and /etc/resolv.conf is empty.

Jun 17 '22 07:06 dxhgq-github

backtrace.log

Jun 17 '22 07:06 dxhgq-github

last week ,i disabled the ipv6 profile ,mv internal-ipv6.xml internal-ipv6.xml.bak ,mv external-ipv6 external-ipv6.bak , but problem is still unsolved。

Jun 17 '22 08:06 dxhgq-github

yesterday ,i disabled the ipv6 on box ,add tow lines : net.ipv6.conf.all.disable_ipv6 =1 net.ipv6.conf.default.disable_ipv6 =1 to /etc/sysctl.conf. by the end of this time ,i have tested for about 30 hours.

Jun 17 '22 08:06 dxhgq-github

This frame 2 is not from a backtrace mentioned in this github issue. From another crash?

Jun 17 '22 08:06 andywolk

@dxhgq-github , https://github.com/signalwire/freeswitch/issues/1659 - you should not open the same issue in two places, sofia-sip and FS, just keep the discussion in one place.

Jun 17 '22 08:06 dragos-oancea

ok , This frame 2 is from another crash, same problem ,above attachment is backtrace

Jun 17 '22 09:06 dxhgq-github

freeswitch freeswitch copied to clipboard

FreeSwitch crashs every day

freeswitch
freeswitch copied to clipboard