roc-toolkit
roc-toolkit copied to clipboard
Published
20 hours ago •
roc-streaming
roc-streaming
SIGABRT pure virtual method called in roc_pipeline benchmark
Reported here: https://github.com/roc-streaming/roc-toolkit/issues/531#issuecomment-1544183101
just tested the roc-toolkit build on develop branch with developer codebook instruction:
scons -Q --build-3rdparty=all \
--enable-werror --enable-debug --enable-tests --enable-benchmarks \
--enable-examples --enable-doxygen test bench
and build was successful and tests got completed. In benchmarks, caught SigABRT
BENCH roc_core
2023-05-11T20:27:38+05:30
Running bin/x86_64-pc-linux-gnu/roc-bench-core
Run on (12 X 4500 MHz CPU s)
CPU Caches:
L1 Data 32 KiB (x6)
L1 Instruction 32 KiB (x6)
L2 Unified 256 KiB (x6)
L3 Unified 12288 KiB (x1)
Load Average: 6.44, 3.69, 1.95
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
----------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------------
BM_MpscQueue/PushBack/iterations:5000000/threads:1 0.027 us 0.027 us 5000000
BM_MpscQueue/PushBack/iterations:5000000/threads:2 0.053 us 0.106 us 10000000
BM_MpscQueue/PushBack/iterations:5000000/threads:4 0.040 us 0.161 us 20000000
BM_MpscQueue/PushBack/iterations:5000000/threads:8 0.034 us 0.275 us 40000000
BM_MpscQueue/PushBack/iterations:5000000/threads:16 0.030 us 0.372 us 80000000
BM_MpscQueue/TryPopFront/1/iterations:5000000 0.089 us 0.089 us 5000000
BM_MpscQueue/TryPopFront/2/iterations:5000000 0.064 us 0.064 us 5000000
BM_MpscQueue/TryPopFront/4/iterations:5000000 0.114 us 0.114 us 5000000
BM_MpscQueue/TryPopFront/8/iterations:5000000 0.181 us 0.181 us 5000000
BM_MpscQueue/TryPopFront/16/iterations:5000000 0.421 us 0.260 us 5000000
BM_MpscQueue/PopFront/1/iterations:5000000 0.082 us 0.082 us 5000000
BM_MpscQueue/PopFront/2/iterations:5000000 0.066 us 0.066 us 5000000
BM_MpscQueue/PopFront/4/iterations:5000000 0.109 us 0.109 us 5000000
BM_MpscQueue/PopFront/8/iterations:5000000 0.201 us 0.201 us 5000000
BM_MpscQueue/PopFront/16/iterations:5000000 0.417 us 0.270 us 5000000
BM_Random_Fast 27.2 ns 27.1 ns 26086588
BENCH roc_pipeline
2023-05-11T20:28:23+05:30
Running bin/x86_64-pc-linux-gnu/roc-bench-pipeline
Run on (12 X 4500 MHz CPU s)
CPU Caches:
L1 Data 32 KiB (x6)
L1 Instruction 32 KiB (x6)
L2 Unified 256 KiB (x6)
L3 Unified 12288 KiB (x1)
Load Average: 7.43, 4.28, 2.23
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
--------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------------------------------------------
BM_PipelineContention/Schedule/iterations:1000000/threads:1 0.131 us 0.130 us 1000000
BM_PipelineContention/Schedule/iterations:1000000/threads:2 0.074 us 0.148 us 2000000
BM_PipelineContention/Schedule/iterations:1000000/threads:4 0.053 us 0.210 us 4000000
BM_PipelineContention/Schedule/iterations:1000000/threads:8 0.035 us 0.280 us 8000000
BM_PipelineContention/Schedule/iterations:1000000/threads:16 0.033 us 0.372 us 16000000
BM_PipelinePeakLoad_NoTasks/iterations:3000/real_time 4999 us 3009 us 3000 fa_avg=1.176 fa_p95=20 fb_avg=2.128 fb_p95=20 pr=0 sc=0 ss=0 t_avg=-nan t_p95=inf tp_frm=-nan tp_plc=-nan
BM_PipelinePeakLoad_PreciseSchedOff/iterations:3000/real_time 4999 us 3017 us 3000 fa_avg=2.344 fa_p95=20 fb_avg=0.58 fb_p95=20 pr=1.535k sc=0 ss=3.002k t_avg=0.096 t_p95=20 tp_frm=0 tp_plc=0.291
pure virtual method called
terminate called without an active exception
ERROR: caught SIGABRT
#1: 0x55e3871fb199 _ZN3roc4core20print_backtrace_safeEv+0x17
#2: 0x55e3871f960a _ZN3roc4core14die_gracefullyEPKcb+0x87
#3: 0x55e3871f90cb _ZN3roc4core12_GLOBAL__N_114signal_handlerEiP9siginfo_tPv+0x31
#4: 0x7f142be4fab0 __sigaction+0x50
#5: 0x7f142be9f26c pthread_key_delete+0x14c
#6: 0x7f142be4fa08 gsignal+0x18
#7: 0x7f142be38538 abort+0xd7
#8: 0x7f142c09ca6f _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold+0x66
#9: 0x7f142c0b011c _ZN10__cxxabiv111__terminateEPFvvE+0xc
#10: 0x7f142c0b0189 _ZSt9terminatev+0x19
#11: 0x7f142c0b0ec7 __cxa_pure_virtual+0x27
#12: 0x55e3871f33c8 _ZN3roc8pipeline12PipelineLoop13process_task_ERNS0_12PipelineTaskEb+0x8e
#13: 0x55e3871f2cf4 _ZN3roc8pipeline12PipelineLoop20maybe_process_tasks_Ev+0xf8
#14: 0x55e3871f2bcc _ZN3roc8pipeline12PipelineLoop13process_tasksEv+0x18
#15: 0x55e3871ec8c7 _ZN3roc8pipeline12_GLOBAL__N_112TestPipeline14do_processing_ERNS_3ctl11ControlTaskE+0x23
#16: 0x55e3871ed4cc _ZN3roc3ctl19ControlTaskExecutorINS_8pipeline12_GLOBAL__N_112TestPipelineEE12execute_taskERNS0_11Co+0x0
#17: 0x55e3871f04f7 _ZN3roc3ctl16ControlTaskQueue13execute_task_ERNS0_11ControlTaskE+0x187
#18: 0x55e3871f07b6 _ZN3roc3ctl16ControlTaskQueue14process_tasks_Ev+0xaa
#19: 0x55e3871ef319 _ZN3roc3ctl16ControlTaskQueue3runEv+0x7d
#20: 0x55e3871fa189 _ZN3roc4core6Thread14thread_runner_EPv+0x23
#21: 0x7f142be9d44b pthread_condattr_setpshared+0x51b
#22: 0x7f142bf20e40 __clone+0x120
scons: *** [bench/roc_pipeline] Error -6
BENCH roc_core
2023-11-06T21:43:07-08:00
Running bin/x86_64-pc-linux-gnu/roc-bench-core
Run on (8 X 3900 MHz CPU s)
CPU Caches:
L1 Data 48 KiB (x4)
L1 Instruction 32 KiB (x4)
L2 Unified 512 KiB (x4)
L3 Unified 8192 KiB (x1)
Load Average: 5.17, 3.75, 2.69
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
----------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------------
BM_MpscQueue/PushBack/iterations:5000000/threads:1 0.021 us 0.021 us 5000000
BM_MpscQueue/PushBack/iterations:5000000/threads:2 0.064 us 0.127 us 10000000
BM_MpscQueue/PushBack/iterations:5000000/threads:4 0.055 us 0.222 us 20000000
BM_MpscQueue/PushBack/iterations:5000000/threads:8 0.031 us 0.245 us 40000000
BM_MpscQueue/PushBack/iterations:5000000/threads:16 0.026 us 0.218 us 80000000
BM_MpscQueue/TryPopFront/1/iterations:5000000 0.058 us 0.058 us 5000000
BM_MpscQueue/TryPopFront/2/iterations:5000000 0.132 us 0.132 us 5000000
BM_MpscQueue/TryPopFront/4/iterations:5000000 0.161 us 0.161 us 5000000
BM_MpscQueue/TryPopFront/8/iterations:5000000 0.160 us 0.150 us 5000000
BM_MpscQueue/TryPopFront/16/iterations:5000000 0.324 us 0.166 us 5000000
BM_MpscQueue/PopFront/1/iterations:5000000 0.080 us 0.080 us 5000000
BM_MpscQueue/PopFront/2/iterations:5000000 0.125 us 0.125 us 5000000
BM_MpscQueue/PopFront/4/iterations:5000000 0.184 us 0.184 us 5000000
BM_MpscQueue/PopFront/8/iterations:5000000 0.196 us 0.180 us 5000000
BM_MpscQueue/PopFront/16/iterations:5000000 0.353 us 0.177 us 5000000
BM_Random_Fast 18.0 ns 18.0 ns 38993209
BENCH roc_pipeline
2023-11-06T21:43:44-08:00
Running bin/x86_64-pc-linux-gnu/roc-bench-pipeline
Run on (8 X 3900 MHz CPU s)
CPU Caches:
L1 Data 48 KiB (x4)
L1 Instruction 32 KiB (x4)
L2 Unified 512 KiB (x4)
L3 Unified 8192 KiB (x1)
Load Average: 6.14, 4.14, 2.86
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
--------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------------------------------------------
BM_PipelineContention/Schedule/iterations:1000000/threads:1 0.171 us 0.170 us 1000000
BM_PipelineContention/Schedule/iterations:1000000/threads:2 0.075 us 0.151 us 2000000
BM_PipelineContention/Schedule/iterations:1000000/threads:4 0.051 us 0.202 us 4000000
BM_PipelineContention/Schedule/iterations:1000000/threads:8 0.030 us 0.227 us 8000000
BM_PipelineContention/Schedule/iterations:1000000/threads:16 0.029 us 0.240 us 16000000
BM_PipelinePeakLoad_NoTasks/iterations:3000/real_time 4999 us 3050 us 3000 fa_avg=1.136 fa_p95=20 fb_avg=2.528 fb_p95=20 pr=0 sc=0 ss=0 t_avg=-nan t_p95=inf tp_frm=-nan tp_plc=-nan
pure virtual method called
terminate called without an active exception
ERROR: caught SIGABRT
#1: 0x557b3a586ec4 _ZN3roc4core20print_backtrace_safeEv+0x1b
#2: 0x557b3a5852f2 _ZN3roc4core14die_gracefullyEPKcb+0x82
#3: 0x557b3a584db5 _ZN3roc4core12_GLOBAL__N_114signal_handlerEiP9siginfo_tPv+0x35
#4: 0x7fe29f6b6420 funlockfile+0x60
#5: 0x7fe29f1a700b gsignal+0xcb
#6: 0x7fe29f186859 abort+0x12b
#7: 0x7fe29f55e8d1 __cxa_throw_bad_array_new_length+0x560
#8: 0x7fe29f56a37c _ZSt17rethrow_exceptionNSt15__exception_ptr13exception_ptrE+0x7c
#9: 0x7fe29f56a3e7 _ZSt9terminatev+0x17
#10: 0x7fe29f56b145 __cxa_pure_virtual+0x25
#11: 0x557b3a57f048 _ZN3roc8pipeline12PipelineLoop13process_task_ERNS0_12PipelineTaskEb+0x92
#12: 0x557b3a57e95e _ZN3roc8pipeline12PipelineLoop20maybe_process_tasks_Ev+0xfc
#13: 0x557b3a57e832 _ZN3roc8pipeline12PipelineLoop13process_tasksEv+0x1c
#14: 0x557b3a57826f _ZN3roc8pipeline12_GLOBAL__N_112TestPipeline14do_processing_ERNS_3ctl11ControlTaskE+0x27
#15: 0x557b3a578f18 _ZN3roc3ctl19ControlTaskExecutorINS_8pipeline12_GLOBAL__N_112TestPipelineEE12execute_taskERNS0_11Co+0x0
#16: 0x557b3a57c065 _ZN3roc3ctl16ControlTaskQueue13execute_task_ERNS0_11ControlTaskE+0x16d
#17: 0x557b3a57c31c _ZN3roc3ctl16ControlTaskQueue14process_tasks_Ev+0xae
#18: 0x557b3a57aed8 _ZN3roc3ctl16ControlTaskQueue3runEv+0x7e
#19: 0x557b3a585e3b _ZN3roc4core6Thread14thread_runner_EPv+0x27
#20: 0x7fe29f6aa609 start_thread+0xd9
#21: 0x7fe29f283133 clone+0x43
Aborted (core dumped)
scons: *** [bench/roc_pipeline] Error 134
I've come across a very similar error while running the same command although the error code is different on Lubuntu 20.04. I figured this would help confirm it is indeed not an issue isolated to just Arch as was mentioned in issue #531
Is this issue still valid? I'm actually seeing a different error.
nolan@nolan-veed:~/nolan-veed/roc-toolkit$ scons -Q --compiler-launcher=ccache --build-3rdparty=all --enable-werror --enable-debug --enable-tests --enable-benchmarks --enable-examples --enable-doxygen bench
BENCH roc_core
2023-12-27T20:51:13+00:00
Running bin/x86_64-pc-linux-gnu/roc-bench-core
Run on (16 X 4900 MHz CPU s)
CPU Caches:
L1 Data 48 KiB (x8)
L1 Instruction 32 KiB (x8)
L2 Unified 1280 KiB (x8)
L3 Unified 24576 KiB (x1)
Load Average: 0.50, 0.76, 1.12
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
----------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------------
BM_MpscQueue/PushBack/iterations:5000000/threads:1 0.027 us 0.027 us 5000000
BM_MpscQueue/PushBack/iterations:5000000/threads:2 0.058 us 0.117 us 10000000
BM_MpscQueue/PushBack/iterations:5000000/threads:4 0.049 us 0.195 us 20000000
BM_MpscQueue/PushBack/iterations:5000000/threads:8 0.043 us 0.341 us 40000000
BM_MpscQueue/PushBack/iterations:5000000/threads:16 0.023 us 0.370 us 80000000
BM_MpscQueue/TryPopFront/1/iterations:5000000 0.060 us 0.060 us 5000000
BM_MpscQueue/TryPopFront/2/iterations:5000000 0.134 us 0.134 us 5000000
BM_MpscQueue/TryPopFront/4/iterations:5000000 0.230 us 0.230 us 5000000
BM_MpscQueue/TryPopFront/8/iterations:5000000 0.319 us 0.319 us 5000000
BM_MpscQueue/TryPopFront/16/iterations:5000000 0.308 us 0.209 us 5000000
BM_MpscQueue/PopFront/1/iterations:5000000 0.084 us 0.084 us 5000000
BM_MpscQueue/PopFront/2/iterations:5000000 0.126 us 0.126 us 5000000
BM_MpscQueue/PopFront/4/iterations:5000000 0.240 us 0.240 us 5000000
BM_MpscQueue/PopFront/8/iterations:5000000 0.370 us 0.370 us 5000000
BM_MpscQueue/PopFront/16/iterations:5000000 0.386 us 0.315 us 5000000
BM_Random_Fast 16.3 ns 16.3 ns 43217227
BENCH roc_pipeline
2023-12-27T20:51:57+00:00
Running bin/x86_64-pc-linux-gnu/roc-bench-pipeline
Run on (16 X 4900 MHz CPU s)
CPU Caches:
L1 Data 48 KiB (x8)
L1 Instruction 32 KiB (x8)
L2 Unified 1280 KiB (x8)
L3 Unified 24576 KiB (x1)
Load Average: 1.62, 1.06, 1.20
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
--------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------------------------------------------
BM_PipelineContention/Schedule/iterations:1000000/threads:1 0.141 us 0.141 us 1000000
BM_PipelineContention/Schedule/iterations:1000000/threads:2 0.075 us 0.150 us 2000000
BM_PipelineContention/Schedule/iterations:1000000/threads:4 0.059 us 0.234 us 4000000
BM_PipelineContention/Schedule/iterations:1000000/threads:8 0.047 us 0.376 us 8000000
BM_PipelineContention/Schedule/iterations:1000000/threads:16 0.035 us 0.524 us 16000000
BM_PipelinePeakLoad_NoTasks/iterations:3000/real_time 15005 us 15003 us 3000 fa_avg=0.216 fa_p95=20 fb_avg=22.8137G fb_p95=inf pr=0 sc=0 ss=0 t_avg=-nan t_p95=inf tp_frm=-nan tp_plc=-nan
BM_PipelinePeakLoad_PreciseSchedOff/iterations:3000/real_time 4999 us 3027 us 3000 fa_avg=2.11 fa_p95=20 fb_avg=7.699 fb_p95=70 pr=1.223k sc=0 ss=3.001k t_avg=1087.58 t_p95=2.87k tp_frm=0 tp_plc=0.302
src/internal_modules/roc_ctl/control_task_queue.cpp:408: error: roc_panic()
ERROR: roc_ctl: control task queue: unexpected already cancelled task in cancel
Backtrace:
#1: 0x562F5D334FFE roc::core::print_backtrace_full()+0x6E
#2: 0x562F5D3333B2 roc::core::die_gracefully(char const*, bool)+0x72
#3: 0x562F5D32E9BD roc::core::panic(char const*, char const*, int, char const*, ...)+0x18D
#4: 0x562F5D327240 roc::ctl::ControlTaskQueue::cancel_task_(roc::ctl::ControlTask&, unsigned int)+0xC0
#5: 0x562F5D326E48 roc::ctl::ControlTaskQueue::renew_scheduling_(roc::ctl::ControlTask&, unsigned int, long, unsigned int)+0x158
#6: 0x562F5D327E4E roc::ctl::ControlTaskQueue::fetch_ready_task_()+0x25E
#7: 0x562F5D327B6F roc::ctl::ControlTaskQueue::fetch_task_()+0x2F
#8: 0x562F5D32688D roc::ctl::ControlTaskQueue::process_tasks_()+0x2D
#9: 0x562F5D3267E3 roc::ctl::ControlTaskQueue::run()+0x83
#10: 0x562F5D333B19 roc::core::Thread::thread_runner_(void*)+0x19
#11: 0x7F0C64E94AC3 pthread_condattr_setpshared+0x513
#12: 0x7F0C64F26660 __xmknodat+0x230
Aborted (core dumped)
scons: *** [bench/roc_pipeline] Error 134
Is this issue still valid? I'm actually seeing a different error.
@nolan-veed this is a different problem I think.
I often see similar panic on CI (where we don't run benchmarks currently, only tests). I guess it's a race either in queue, or in both tests & benchmarks. Probably it's related to https://github.com/roc-streaming/roc-toolkit/issues/435, but I'm not sure. Never had time to debug it yet..