Eval: HIP: Llama-server multi-instance lockup

Open pl752 opened this issue 8 months ago • 3 comments

Follow-up on the https://github.com/ggml-org/llama.cpp/issues/12991

According to rocgdb backtrace, threads that are working with gpus are stuck somewhere in the libhsa-runtime

Thread 2 (Thread 0x7fffd7fff6c0 (LWP 11602) "llama-server"):
#0  __GI___ioctl (fd=3, request=3222817548) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x00007fffd8549400 in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libhsa-runtime64.so.1
#2  0x00007fffd8541f1f in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libhsa-runtime64.so.1
#3  0x00007fffd84be632 in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libhsa-runtime64.so.1
#4  0x00007fffd84a1aee in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libhsa-runtime64.so.1
#5  0x00007fffd8439241 in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libhsa-runtime64.so.1
#6  0x00007ffff749caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#7  0x00007ffff7529c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 1 (Thread 0x7fffec12f840 (LWP 11595) "llama-server"):
#0  0x00007fffd847cbf4 in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libhsa-runtime64.so.1
#1  0x00007fffd847ca3e in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libhsa-runtime64.so.1
#2  0x00007fffd8470c81 in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libhsa-runtime64.so.1
#3  0x00007fffeabb9ebb in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libamdhip64.so.6
#4  0x00007fffeaba6087 in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libamdhip64.so.6
#5  0x00007fffeaac7618 in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libamdhip64.so.6
#6  0x00007fffeaac7cb8 in ?? () from /opt/rocm-6.4.0/lib/llvm/bin/../../../lib/libamdhip64.so.6
#7  0x00007fffefafa63e in ggml_backend_cuda_buffer_set_tensor(ggml_backend_buffer*, ggml_tensor*, void const*, unsigned long, unsigned long) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#8  0x00007ffff7c88189 in ggml_backend_sched_graph_compute_async () from /home/pl752/llama.cpp/build/bin/libggml-base.so
#9  0x00007ffff7db4c51 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/pl752/llama.cpp/build/bin/libllama.so
#10 0x00007ffff7db7038 in llama_context::decode(llama_batch&) () from /home/pl752/llama.cpp/build/bin/libllama.so
#11 0x00007ffff7db843f in llama_decode () from /home/pl752/llama.cpp/build/bin/libllama.so
#12 0x00005555555ff4d9 in server_context::update_slots() ()
#13 0x00005555555c9dd1 in server_queue::start_loop() ()
#14 0x00005555555972b1 in main ()

(Sorry for not using the template)

Apr 24 '25 15:04 pl752

Again since a process (even a misbehaving one) stalling a different process should be impossible. This is by definition at least primarily not a llamacpp bug. You should report it to amd instead.

The backtrace is also not useful without debug symbols for the rocm components.

Apr 24 '25 18:04 IMbackK

Okay, thank you for the reply. I will try to investigate further and give updates on this weird behaviour.

Apr 25 '25 02:04 pl752

Here are some backtraces with debug symbols for amdhsa and hip runtime and with RelWithDbg build for some of the interesting threads

Rocgdb bt output

slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 24301
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.084276
^C
Thread 1 "llama-server" received signal SIGINT, Interrupt.
0x00007fffd627cbde in rocr::__rdtsc () at /usr/lib/gcc/x86_64-linux-gnu/13/include/ia32intrin.h:114
114       return __builtin_ia32_rdtsc ();
(gdb) thread apply all bt

Thread 42 (Thread 0x7ffc33ff76c0 (LWP 21847) "llama-server"):
#0  0x00007ffff664c6ee in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff6649779 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 41 (Thread 0x7ffc347f86c0 (LWP 21846) "llama-server"):
#0  0x00007ffff664c6ee in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff6649779 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 40 (Thread 0x7ffc34ff96c0 (LWP 21845) "llama-server"):
#0  0x00007ffff664c6ee in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff6649779 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 39 (Thread 0x7ffc357fa6c0 (LWP 21844) "llama-server"):
#0  0x00007ffff664c6ee in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff6649779 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 38 (Thread 0x7ffc35ffb6c0 (LWP 21843) "llama-server"):
#0  0x00007ffff664c6ee in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff6649779 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 37 (Thread 0x7ffc367fc6c0 (LWP 21842) "llama-server"):
#0  0x00007ffff664c6ee in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff6649779 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 36 (Thread 0x7ffc36ffd6c0 (LWP 21841) "llama-server"):
#0  0x00007ffff664c6ee in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff6649779 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 35 (Thread 0x7ffc377fe6c0 (LWP 21840) "llama-server"):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x00007ffff664c6ee in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff6649779 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 34 (Thread 0x7ffc37fff6c0 (LWP 21839) "llama-server"):
#0  0x00007ffff664c6ee in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff6649779 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 33 (Thread 0x7ffc3cbff6c0 (LWP 21838) "llama-server"):
#0  0x00007ffff664c6ee in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff6649779 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 32 (Thread 0x7ffc3edff6c0 (LWP 21837) "llama-server"):
#0  0x00007ffff664c6ee in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff6649779 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 31 (Thread 0x7fffd65bf6c0 (LWP 21836) "llama-server"):
#0  __GI___ioctl (fd=fd@entry=3, request=request@entry=3222817548) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x00007fffd6349400 in hsakmt_ioctl (fd=3, request=request@entry=3222817548, arg=arg@entry=0x7fffd65b5b10) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/libhsakmt.c:13
#2  0x00007fffd6341f1f in hsaKmtWaitOnMultipleEvents_Ext (event_age=0x7fffd65b5c18, Milliseconds=15, WaitOnAll=true, NumEvents=<optimized out>, Events=0x7fffd65b5bc8) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:416
#3  hsaKmtWaitOnMultipleEvents_Ext (Events=0x7fffd65b5bc8, NumEvents=1, WaitOnAll=<optimized out>, Milliseconds=15, event_age=0x7fffd65b5c18) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:385
#4  0x00007fffd63426db in hsaKmtWaitOnEvent_Ext (Event=<optimized out>, Milliseconds=<optimized out>, event_age=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:231
#5  0x00007fffd627cd3b in rocr::core::InterruptSignal::WaitRelaxed (this=0x5555668958a0, condition=HSA_SIGNAL_CONDITION_NE, compare_value=1, timeout=<optimized out>, wait_hint=HSA_WAIT_STATE_BLOCKED) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/interrupt_signal.cpp:197
#6  0x00007fffd627ca3e in rocr::core::InterruptSignal::WaitAcquire (this=<optimized out>, condition=<optimized out>, compare_value=<optimized out>, timeout=<optimized out>, wait_hint=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/interrupt_signal.cpp:205
#7  0x00007fffd6270c81 in rocr::HSA::hsa_signal_wait_scacquire (hsa_signal=..., condition=HSA_SIGNAL_CONDITION_NE, compare_value=1, timeout_hint=16000000, wait_state_hint=HSA_WAIT_STATE_BLOCKED) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/hsa.cpp:1249
#8  0x00007fffe89f7d55 in amd::HostcallListener::consumePackets (this=0x555567386c30) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/device/devhostcall.cpp:307
#9  0x00007fffe8657412 in amd::Thread::main (this=this@entry=0x555567386cc8) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/thread/thread.cpp:93
#10 0x00007fffe899bfa6 in amd::Thread::entry (thread=0x555567386cc8) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/os/os_posi--Type <RET> for more, q to quit, c to continue without paging--
x.cpp:351
#11 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#12 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 29 (Thread 0x7ffd737ff6c0 (LWP 21830) "llama-server"):
#0  __GI___ioctl (fd=fd@entry=3, request=request@entry=3222817548) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x00007fffd6349400 in hsakmt_ioctl (fd=3, request=request@entry=3222817548, arg=arg@entry=0x7ffd737f5c30) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/libhsakmt.c:13
#2  0x00007fffd6341f1f in hsaKmtWaitOnMultipleEvents_Ext (event_age=0x7ffd6c0019d0, Milliseconds=4294967294, WaitOnAll=false, NumEvents=<optimized out>, Events=0x7ffd6c001920) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:416
#3  hsaKmtWaitOnMultipleEvents_Ext (Events=0x7ffd6c001920, NumEvents=2, WaitOnAll=<optimized out>, Milliseconds=4294967294, event_age=0x7ffd6c0019d0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:385
#4  0x00007fffd62a175c in operator() (__closure=<synthetic pointer>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/runtime.cpp:1601
#5  rocr::core::Runtime::AsyncEventsLoop (_eventsInfo=0x5555560bd8b8) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/runtime.cpp:1691
#6  0x00007fffd6239241 in rocr::os::ThreadTrampoline (arg=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/util/lnx/os_linux.cpp:86
#7  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#8  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 28 (Thread 0x7ffdb07f86c0 (LWP 21829) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 27 (Thread 0x7ffdb0ff96c0 (LWP 21828) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@--Type <RET> for more, q to quit, c to continue without paging--
entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 26 (Thread 0x7ffdb17fa6c0 (LWP 21827) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 25 (Thread 0x7ffdb1ffb6c0 (LWP 21826) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
--Type <RET> for more, q to quit, c to continue without paging--
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 24 (Thread 0x7ffdb27fc6c0 (LWP 21825) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 23 (Thread 0x7ffdb2ffd6c0 (LWP 21824) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
--Type <RET> for more, q to quit, c to continue without paging--
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 22 (Thread 0x7ffdb37fe6c0 (LWP 21823) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 21 (Thread 0x7ffdb3fff6c0 (LWP 21822) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 20 (Thread 0x7ffdb8ff96c0 (LWP 21821) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
--Type <RET> for more, q to quit, c to continue without paging--
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 19 (Thread 0x7ffdb97fa6c0 (LWP 21820) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 18 (Thread 0x7ffdb9ffb6c0 (LWP 21819) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
--Type <RET> for more, q to quit, c to continue without paging--
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 17 (Thread 0x7ffdba7fc6c0 (LWP 21818) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 16 (Thread 0x7ffdbaffd6c0 (LWP 21817) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
--Type <RET> for more, q to quit, c to continue without paging--
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 15 (Thread 0x7ffdbb7fe6c0 (LWP 21816) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 14 (Thread 0x7ffdbbfff6c0 (LWP 21815) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 13 (Thread 0x7ffdc0ffa6c0 (LWP 21814) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:57
--Type <RET> for more, q to quit, c to continue without paging--
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 12 (Thread 0x7ffdc17fb6c0 (LWP 21813) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7ffdc17f0070, op=137, expected=0, futex_word=0x7fffffffd8d8) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7ffdc17f0070, clockid=0, expected=0, futex_word=0x7fffffffd8d8) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fffffffd8d8, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7ffdc17f0070, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709c116 in __pthread_cond_wait_common (abstime=<optimized out>, clockid=<optimized out>, mutex=0x7fffffffd888, cond=0x7fffffffd8b0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_clockwait64 (abstime=<optimized out>, clockid=<optimized out>, mutex=0x7fffffffd888, cond=0x7fffffffd8b0) at ./nptl/pthread_cond_wait.c:691
#5  ___pthread_cond_clockwait64 (cond=0x7fffffffd8b0, mutex=0x7fffffffd888, clockid=<optimized out>, abstime=<optimized out>) at ./nptl/pthread_cond_wait.c:679
#6  0x000055555560e9c9 in std::__condvar::wait_until(std::mutex&, int, timespec&) ()
#7  0x00005555556b1d3c in std::cv_status std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#8  0x00005555556940b7 in std::cv_status std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#9  0x0000555555671daa in std::cv_status std::condition_variable::wait_for<long, std::ratio<1l, 1l> >(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1l> > const&) ()
#10 0x0000555555649a5f in server_response::recv_with_timeout(std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&, int) ()
#11 0x000055555565261a in server_context::receive_cmpl_results_stream(std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&, std::function<bool (std::unique_ptr<server_task_result, std::default_delete<server_task_result> >&)> const&, std::function<void (nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>)> const&, std::function<bool ()> const&) ()
#12 0x00005555555f5b68 in main::{lambda(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type)#1}::operator()(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type) const::{lambda(unsigned long, httplib::DataSink&)#1}::operator()(unsigned long, httplib::DataSink&) const ()
#13 0x0000555555609e54 in bool std::__invoke_impl<bool, main::{lambda(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std--Type <RET> for more, q to quit, c to continue without paging--
::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type)#1}::operator()(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type) const::{lambda(unsigned long, httplib::DataSink&)#1}&, unsigned long, httplib::DataSink&>(std::__invoke_other, main::{lambda(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type)#1}::operator()(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type) const::{lambda(unsigned long, httplib::DataSink&)#1}&, unsigned long&&, httplib::DataSink&) ()
#14 0x000055555560661b in std::enable_if<is_invocable_r_v<bool, main::{lambda(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type)#1}::operator()(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type) const::{lambda(unsigned long, httplib::DataSink&)#1}&, unsigned long, httplib::DataSink&>, bool>::type std::__invoke_r<bool, main::{lambda(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type)#1}::operator()(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type) const::{lambda(unsigned long, httplib::DataSink&)#1}&, unsigned long, httplib::DataSink&>(main::{lambda(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type)#1}::operator()(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type) const::{lambda(unsigned long, httplib::DataSink&)#1}&, unsigned long&&, httplib::DataSink&) ()
#15 0x00005555556033a9 in std::_Function_handler<bool (unsigned long, httplib::DataSink&), main::{lambda(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type)#1}::operator()(server_task_type, nlohmann::json_abi_v3_11_3::basic_json<nlohmann::json_abi_v3_11_3::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void>&, std::function<bool ()>, httplib::Response&, oaicompat_type) const::{lambda(unsigned long, httplib::DataSink&)#1}>::_M_invoke(std::_Any_data const&, unsigned long&&, httplib::DataSink&) ()
#16 0x00005555556610b1 in std::function<bool (unsigned long, httplib::DataSink&)>::operator()(unsigned long, httplib::DataSink&) const ()
#17 0x00005555556199e3 in httplib::detail::ContentProviderAdapter::operator()(unsigned long, unsigned long, httplib::DataSink&) ()
#18 0x00005555556d369a in bool std::__invoke_impl<bool, httplib::detail::ContentProviderAdapter&, unsigned long, unsigned long, httplib::DataSink&>(std::__invoke_other, httplib::detail::ContentProviderAdapter&, unsigned long&&, unsigned long&&, httplib::DataSink&) ()
#19 0x00005555556beadf in std::enable_if<is_invocable_r_v<bool, httplib::detail::ContentProviderAdapter&, unsigned long, unsigned long, httplib::DataSink&>, bool>::type std::__invoke_r<bool, httplib::detail::ContentProviderAdapter&, unsigned long, unsigned long, httplib::DataSink&>(httplib::detail::ContentProviderAdapter&, unsigned long&&, unsigned long&&, httplib::DataSink&) ()
#20 0x00005555556a1430 in std::_Function_handler<bool (unsigned long, unsigned long, httplib::DataSink&), httplib::detail::ContentProviderAdapter>::_M_invoke(std::_Any_data const&, unsigned long&&, unsigned long&&, httplib::DataSink&) ()
#21 0x0000555555683690 in std::function<bool (unsigned long, unsigned long, httplib::DataSink&)>::operator()(unsigned long, unsigned long, httplib::DataSink&) const ()
#22 0x00005555556833c2 in bool httplib::detail::write_content_chunked<httplib::Server::write_content_with_provider(httplib::Stream&, httplib::Request const&, httplib::Response&, std::_--Type <RET> for more, q to quit, c to continue without paging--
_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}, httplib::detail::compressor>(httplib::Stream&, std::function<bool (unsigned long, unsigned long, httplib::DataSink&)> const&, httplib::Server::write_content_with_provider(httplib::Stream&, httplib::Request const&, httplib::Response&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1} const&, httplib::detail::compressor&, httplib::Error&) ()
#23 0x0000555555663319 in bool httplib::detail::write_content_chunked<httplib::Server::write_content_with_provider(httplib::Stream&, httplib::Request const&, httplib::Response&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}, httplib::detail::compressor>(httplib::Stream&, std::function<bool (unsigned long, unsigned long, httplib::DataSink&)> const&, httplib::Server::write_content_with_provider(httplib::Stream&, httplib::Request const&, httplib::Response&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1} const&, httplib::detail::compressor&) ()
#24 0x000055555561eda1 in httplib::Server::write_content_with_provider(httplib::Stream&, httplib::Request const&, httplib::Response&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#25 0x000055555561e751 in httplib::Server::write_response_core(httplib::Stream&, bool, httplib::Request const&, httplib::Response&, bool) ()
#26 0x000055555561dbb2 in httplib::Server::write_response_with_content(httplib::Stream&, bool, httplib::Request const&, httplib::Response&) ()
#27 0x0000555555623f81 in httplib::Server::process_request(httplib::Stream&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool, bool&, std::function<void (httplib::Request&)> const&) ()
#28 0x00005555556248aa in httplib::Server::process_and_close_socket(int)::{lambda(httplib::Stream&, bool, bool&)#1}::operator()(httplib::Stream&, bool, bool&) const ()
#29 0x0000555555665b2b in httplib::detail::process_server_socket<httplib::Server::process_and_close_socket(int)::{lambda(httplib::Stream&, bool, bool&)#1}>(std::atomic<int> const&, int, unsigned long, long, long, long, long, long, httplib::Server::process_and_close_socket(int)::{lambda(httplib::Stream&, bool, bool&)#1})::{lambda(bool, bool&)#1}::operator()(bool, bool&) const ()
#30 0x0000555555685cd8 in bool httplib::detail::process_server_socket_core<httplib::detail::process_server_socket<httplib::Server::process_and_close_socket(int)::{lambda(httplib::Stream&, bool, bool&)#1}>(std::atomic<int> const&, int, unsigned long, long, long, long, long, long, httplib::Server::process_and_close_socket(int)::{lambda(httplib::Stream&, bool, bool&)#1})::{lambda(bool, bool&)#1}>(std::atomic<int> const&, int, unsigned long, long, httplib::detail::process_server_socket<httplib::Server::process_and_close_socket(int)::{lambda(httplib::Stream&, bool, bool&)#1}>(std::atomic<int> const&, int, unsigned long, long, long, long, long, long, httplib::Server::process_and_close_socket(int)::{lambda(httplib::Stream&, bool, bool&)#1})::{lambda(bool, bool&)#1}) ()
#31 0x0000555555665c2b in bool httplib::detail::process_server_socket<httplib::Server::process_and_close_socket(int)::{lambda(httplib::Stream&, bool, bool&)#1}>(std::atomic<int> const&, int, unsigned long, long, long, long, long, long, httplib::Server::process_and_close_socket(int)::{lambda(httplib::Stream&, bool, bool&)#1}) ()
#32 0x0000555555624a95 in httplib::Server::process_and_close_socket(int) ()
#33 0x0000555555620888 in httplib::Server::listen_internal()::{lambda()#2}::operator()() const ()
#34 0x00005555556c2134 in void std::__invoke_impl<void, httplib::Server::listen_internal()::{lambda()#2}&>(std::__invoke_other, httplib::Server::listen_internal()::{lambda()#2}&) ()
#35 0x00005555556a5472 in std::enable_if<is_invocable_r_v<void, httplib::Server::listen_internal()::{lambda()#2}&>, void>::type std::__invoke_r<void, httplib::Server::listen_internal()::{lambda()#2}&>(httplib::Server::listen_internal()::{lambda()#2}&) ()
#36 0x0000555555684c61 in std::_Function_handler<void (), httplib::Server::listen_internal()::{lambda()#2}>::_M_invoke(std::_Any_data const&) ()
#37 0x000055555565bb08 in std::function<void ()>::operator()() const ()
#38 0x000055555561054b in httplib::ThreadPool::worker::operator()() ()
#39 0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#40 0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#41 0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#42 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#43 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#44 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#45 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#46 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 11 (Thread 0x7ffdc1ffc6c0 (LWP 21812) "llama-server"):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 10 (Thread 0x7ffdc27fd6c0 (LWP 21811) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 9 (Thread 0x7ffdc2ffe6c0 (LWP 21810) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 8 (Thread 0x7ffdc37ff6c0 (LWP 21809) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 7 (Thread 0x7fffcde536c0 (LWP 21808) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
--Type <RET> for more, q to quit, c to continue without paging--
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 6 (Thread 0x7fffcf7fe6c0 (LWP 21807) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffdb4000be4) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffdb4000be4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffdb4000be8, cond=0x7ffdb4000bb8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7ffdb4000bb8, mutex=0x7ffdb4000be8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555565c52d in void std::condition_variable::wait<httplib::ThreadPool::worker::operator()()::{lambda()#1}>(std::unique_lock<std::mutex>&, httplib::ThreadPool::worker::operator()()::{lambda()#1}) ()
#6  0x000055555561046e in httplib::ThreadPool::worker::operator()() ()
#7  0x0000555555734632 in void std::__invoke_impl<void, httplib::ThreadPool::worker>(std::__invoke_other, httplib::ThreadPool::worker&&) ()
#8  0x00005555557345ed in std::__invoke_result<httplib::ThreadPool::worker>::type std::__invoke<httplib::ThreadPool::worker>(httplib::ThreadPool::worker&&) ()
#9  0x000055555573458e in void std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x00005555557344a4 in std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> >::operator()() ()
#11 0x00005555557343c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 5 (Thread 0x7fffcffff6c0 (LWP 21806) "llama-server"):
#0  0x00007ffff712b8e4 in accept4 (fd=10, addr=..., addr_len=0x0, flags=524288) at ../sysdeps/unix/sysv/linux/accept4.c:31
#1  0x0000555555620a05 in httplib::Server::listen_internal() ()
#2  0x000055555561cf54 in httplib::Server::listen_after_bind() ()
#3  0x00005555555fcd73 in main::{lambda()#3}::operator()() const ()
#4  0x000055555560cf0e in void std::__invoke_impl<void, main::{lambda()#3}>(std::__invoke_other, main::{lambda()#3}&&) ()
#5  0x000055555560ced1 in std::__invoke_result<main::{lambda()#3}>::type std::__invoke<main::{lambda()#3}>(main::{lambda()#3}&&) ()
#6  0x000055555560ce7e in void std::thread::_Invoker<std::tuple<main::{lambda()#3}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#7  0x000055555560ce52 in std::thread::_Invoker<std::tuple<main::{lambda()#3}> >::operator()() ()
#8  0x000055555560ce36 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::{lambda()#3}> > >::_M_run() ()
#9  0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#11 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 4 (Thread 0x7fffd49ff6c0 (LWP 21805) "llama-server"):
#0  0x00007ffff7098d71 in __futex_abstimed_wait_common64 (private=32765, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555b02f98 <common_log_main()::log+88>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=32765, abstime=0x0, clockid=0, expected=0, futex_word=0x555555b02f98 <common_log_main()::log+88>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555b02f98 <common_log_main()::log+88>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime--Type <RET> for more, q to quit, c to continue without paging--
@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff709b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555555b02f40 <common_log_main()::log>, cond=0x555555b02f70 <common_log_main()::log+48>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x555555b02f70 <common_log_main()::log+48>, mutex=0x555555b02f40 <common_log_main()::log>) at ./nptl/pthread_cond_wait.c:627
#5  0x000055555589d8b3 in void std::condition_variable::wait<common_log::resume()::{lambda()#1}::operator()() const::{lambda()#1}>(std::unique_lock<std::mutex>&, common_log::resume()::{lambda()#1}::operator()() const::{lambda()#1}) ()
#6  0x000055555589cd4b in common_log::resume()::{lambda()#1}::operator()() const ()
#7  0x000055555589f390 in void std::__invoke_impl<void, common_log::resume()::{lambda()#1}>(std::__invoke_other, common_log::resume()::{lambda()#1}&&) ()
#8  0x000055555589f34b in std::__invoke_result<common_log::resume()::{lambda()#1}>::type std::__invoke<common_log::resume()::{lambda()#1}>(common_log::resume()::{lambda()#1}&&) ()
#9  0x000055555589f2ec in void std::thread::_Invoker<std::tuple<common_log::resume()::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#10 0x000055555589f2bc in std::thread::_Invoker<std::tuple<common_log::resume()::{lambda()#1}> >::operator()() ()
#11 0x000055555589f29c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<common_log::resume()::{lambda()#1}> > >::_M_run() ()
#12 0x00007ffff74ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 2 (Thread 0x7fffd5dff6c0 (LWP 21803) "llama-server"):
#0  __GI___ioctl (fd=fd@entry=3, request=request@entry=3222817548) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x00007fffd6349400 in hsakmt_ioctl (fd=3, request=request@entry=3222817548, arg=arg@entry=0x7fffd5df5b80) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/libhsakmt.c:13
#2  0x00007fffd6341f1f in hsaKmtWaitOnMultipleEvents_Ext (event_age=0x7fffd5df5c30, Milliseconds=4294967295, WaitOnAll=false, NumEvents=<optimized out>, Events=0x5555560cf130) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:416
#3  hsaKmtWaitOnMultipleEvents_Ext (Events=0x5555560cf130, NumEvents=3, WaitOnAll=<optimized out>, Milliseconds=4294967295, event_age=0x7fffd5df5c30) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:385
#4  0x00007fffd62be632 in rocr::core::Signal::WaitAnyExceptions (signal_count=<optimized out>, hsa_signals=0x7fffd0000bb0, conds=<optimized out>, values=<optimized out>, satisfying_value=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/signal.cpp:428
#5  0x00007fffd62a1aee in rocr::core::Runtime::AsyncEventsLoop (_eventsInfo=0x5555560bda40) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/runtime.cpp:1612
#6  0x00007fffd6239241 in rocr::os::ThreadTrampoline (arg=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/util/lnx/os_linux.cpp:86
#7  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#8  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 1 (Thread 0x7fffe9f08840 (LWP 21796) "llama-server"):
#0  0x00007fffd627cbde in rocr::__rdtsc () at /usr/lib/gcc/x86_64-linux-gnu/13/include/ia32intrin.h:114
#1  rocr::timer::fast_clock::raw_now () at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/util/timer.h:149
#2  rocr::timer::fast_clock::now () at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/util/timer.h:140
#3  rocr::core::InterruptSignal::WaitRelaxed (this=0x5555672b8210, condition=HSA_SIGNAL_CONDITION_LT, compare_value=1, timeout=<optimized out>, wait_hint=HSA_WAIT_STATE_ACTIVE) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/interrupt_signal.cpp:166
#4  0x00007fffd627ca3e in rocr::core::InterruptSignal::WaitAcquire (this=<optimized out>, condition=<optimized out>, compare_value=<optimized out>, timeout=<optimized out>, wait_hint=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/interrupt_signal.cpp:205
#5  0x00007fffd6270c81 in rocr::HSA::hsa_signal_wait_scacquire (hsa_signal=..., condition=condition@entry=HSA_SIGNAL_CONDITION_LT, compare_value=compare_value@entry=1, timeout_hint=tim--Type <RET> for more, q to quit, c to continue without paging--
eout_hint@entry=18446744073709551615, wait_state_hint=wait_state_hint@entry=HSA_WAIT_STATE_ACTIVE) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/hsa.cpp:1249
#6  0x00007fffd6243030 in rocr::AMD::BlitKernel::SubmitLinearCopyCommand (this=0x5555668e2b60, dst=0x7ffc2e200000, src=0x7ffc2de00000, size=2665873) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/amd_blit_kernel.cpp:1015
#7  0x00007fffd6264ee8 in rocr::(anonymous namespace)::RegionMemory::Freeze (this=0x55556a4b89e0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/amd_loader_context.cpp:354
#8  0x00007fffd62c6bd2 in rocr::amd::hsa::loader::Segment::Freeze (this=0x55556a4b8ac0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/loader/executable.cpp:706
#9  rocr::amd::hsa::loader::ExecutableImpl::Freeze (this=0x55556a4b8770, options=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/loader/executable.cpp:1946
#10 0x00007fffd62c7d2b in rocr::amd::hsa::loader::AmdHsaCodeLoader::FreezeExecutable (this=0x5555560ba170, executable=0x55556a4b8770, options=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/loader/executable.cpp:237
#11 0x00007fffd627345b in rocr::HSA::hsa_executable_freeze (executable=..., options=0x0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/hsa.cpp:2335
#12 0x00007fffe89ca423 in amd::roc::LightningProgram::setKernels (this=0x5555626f5480, binary=0x7ffff3547000, binSize=<optimized out>, fdesc=<optimized out>, foffset=<optimized out>, uri=...) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/device/rocm/rocprogram.cpp:328
#13 0x00007fffe897e8aa in amd::device::Program::loadLC (this=this@entry=0x5555626f5480) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/device/devprogram.cpp:1877
#14 0x00007fffe897e8e3 in amd::device::Program::load (this=this@entry=0x5555626f5480) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/device/devprogram.cpp:1888
#15 0x00007fffe89adb23 in amd::Program::load (this=0x5555562155b0, devices=...) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/platform/program.cpp:614
#16 0x00007fffe86b3fe8 in hip::FatBinaryInfo::BuildProgram (this=<optimized out>, device_id=device_id@entry=0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_fatbin.cpp:722
#17 0x00007fffe86ba5fb in hip::Function::getStatFunc (this=0x5555560970f0, hfunc=hfunc@entry=0x7fffffff7540, deviceId=deviceId@entry=0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_global.cpp:165
#18 0x00007fffe865ae9c in hip::StatCO::getStatFunc (this=0x555555b042e8, hfunc=hfunc@entry=0x7fffffff7540, hostFunction=hostFunction@entry=0x7fffedaffc20 <void mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int)>, deviceId=deviceId@entry=0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_code_object.cpp:1389
#19 0x00007fffe88a131d in hip::PlatformState::getStatFunc (this=<optimized out>, hfunc=hfunc@entry=0x7fffffff7540, hostFunction=hostFunction@entry=0x7fffedaffc20 <void mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int)>, deviceId=deviceId@entry=0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_platform.cpp:931
#20 0x00007fffe88a1410 in hip::ihipLaunchKernel (hostFunction=0x7fffedaffc20 <void mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int)>, gridDim=..., blockDim=..., args=0x7fffffff7cf0, sharedMemBytes=30336, stream=<optimized out>, startEvent=0x0, stopEvent=0x0, flags=0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_platform.cpp:625
#21 0x00007fffe88607ee in hip::hipLaunchKernel_common (hostFunction=<optimized out>, hostFunction@entry=0x7fffedaffc20 <void mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int)>, gridDim=..., blockDim=..., args=<optimized out>, args@entry=0x7fffffff7cf0, sharedMemBytes=<optimized out>, stream=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_module.cpp:738
#22 0x00007fffe88725e1 in hip::hipLaunchKernel (hostFunction=0x7fffedaffc20 <void mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int)>, gridDim=..., blockDim=..., args=<optimized out>, sharedMemBytes=<optimized out>, stream=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_module.cpp:745
#23 0x00007fffeda7f53c in void __device_stub__mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#24 0x00007fffeda79297 in void launch_mul_mat_q<(ggml_type)3, 64>(ggml_backend_cuda_context&, mmq_args const&, ihipStream_t*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
--Type <RET> for more, q to quit, c to continue without paging--
#25 0x00007fffeda82d40 in void mul_mat_q_case<(ggml_type)3>(ggml_backend_cuda_context&, mmq_args const&, ihipStream_t*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#26 0x00007fffed8f03d4 in ggml_cuda_op_mul_mat_q(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*, char const*, float const*, char const*, float*, long, long, long, long, ihipStream_t*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#27 0x00007fffed8e5669 in ggml_cuda_op_mul_mat(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*, void (*)(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*, char const*, float const*, char const*, float*, long, long, long, long, ihipStream_t*), void (*)(float const*, void*, ggml_type, long, long, long, long, long, long, long, long, ihipStream_t*)) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#28 0x00007fffed8e1aa2 in ggml_cuda_mul_mat(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#29 0x00007fffed8e1388 in ggml_cuda_compute_forward(ggml_backend_cuda_context&, ggml_tensor*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#30 0x00007fffed8e0f5b in evaluate_and_capture_cuda_graph(ggml_backend_cuda_context*, ggml_cgraph*, bool&, bool&, bool&) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#31 0x00007fffed8e0b6c in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#32 0x00007ffff7731e9c in ggml_backend_graph_compute_async () from /home/pl752/llama.cpp/build/bin/libggml-base.so
#33 0x00007ffff7735fd4 in ggml_backend_sched_compute_splits(ggml_backend_sched*) () from /home/pl752/llama.cpp/build/bin/libggml-base.so
#34 0x00007ffff7736c57 in ggml_backend_sched_graph_compute_async () from /home/pl752/llama.cpp/build/bin/libggml-base.so
#35 0x00007ffff7b5ce17 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/pl752/llama.cpp/build/bin/libllama.so
#36 0x00007ffff7b5b3a4 in llama_context::decode(llama_batch&) () from /home/pl752/llama.cpp/build/bin/libllama.so
#37 0x00007ffff7b6074e in llama_decode () from /home/pl752/llama.cpp/build/bin/libllama.so
#38 0x000055555565606f in server_context::update_slots() ()
#39 0x00005555555fcdc7 in main::{lambda()#4}::operator()() const ()
#40 0x000055555560b384 in void std::__invoke_impl<void, main::{lambda()#4}&>(std::__invoke_other, main::{lambda()#4}&) ()
#41 0x0000555555609254 in std::enable_if<is_invocable_r_v<void, main::{lambda()#4}&>, void>::type std::__invoke_r<void, main::{lambda()#4}&>(main::{lambda()#4}&) ()
#42 0x0000555555605483 in std::_Function_handler<void (), main::{lambda()#4}>::_M_invoke(std::_Any_data const&) ()
#43 0x000055555565bb08 in std::function<void ()>::operator()() const ()
#44 0x0000555555648c41 in server_queue::start_loop() ()
#45 0x00005555555ff634 in main ()
(gdb)

Thread 1 bt full

(gdb) thread apply 1 bt

Thread 1 (Thread 0x7fffe9f08840 (LWP 21796) "llama-server"):
#0  0x00007fffd627cbde in rocr::__rdtsc () at /usr/lib/gcc/x86_64-linux-gnu/13/include/ia32intrin.h:114
#1  rocr::timer::fast_clock::raw_now () at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/util/timer.h:149
#2  rocr::timer::fast_clock::now () at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/util/timer.h:140
#3  rocr::core::InterruptSignal::WaitRelaxed (this=0x5555672b8210, condition=HSA_SIGNAL_CONDITION_LT, compare_value=1, timeout=<optimized out>, wait_hint=HSA_WAIT_STATE_ACTIVE) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/interrupt_signal.cpp:166
#4  0x00007fffd627ca3e in rocr::core::InterruptSignal::WaitAcquire (this=<optimized out>, condition=<optimized out>, compare_value=<optimized out>, timeout=<optimized out>, wait_hint=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/interrupt_signal.cpp:205
#5  0x00007fffd6270c81 in rocr::HSA::hsa_signal_wait_scacquire (hsa_signal=..., condition=condition@entry=HSA_SIGNAL_CONDITION_LT, compare_value=compare_value@entry=1, timeout_hint=timeout_hint@entry=18446744073709551615, wait_state_hint=wait_state_hint@entry=HSA_WAIT_STATE_ACTIVE) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/hsa.cpp:1249
#6  0x00007fffd6243030 in rocr::AMD::BlitKernel::SubmitLinearCopyCommand (this=0x5555668e2b60, dst=0x7ffc2e200000, src=0x7ffc2de00000, size=2665873) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/amd_blit_kernel.cpp:1015
#7  0x00007fffd6264ee8 in rocr::(anonymous namespace)::RegionMemory::Freeze (this=0x55556a4b89e0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/amd_loader_context.cpp:354
#8  0x00007fffd62c6bd2 in rocr::amd::hsa::loader::Segment::Freeze (this=0x55556a4b8ac0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/loader/executable.cpp:706
#9  rocr::amd::hsa::loader::ExecutableImpl::Freeze (this=0x55556a4b8770, options=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/loader/executable.cpp:1946
#10 0x00007fffd62c7d2b in rocr::amd::hsa::loader::AmdHsaCodeLoader::FreezeExecutable (this=0x5555560ba170, executable=0x55556a4b8770, options=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/loader/executable.cpp:237
#11 0x00007fffd627345b in rocr::HSA::hsa_executable_freeze (executable=..., options=0x0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/hsa.cpp:2335
#12 0x00007fffe89ca423 in amd::roc::LightningProgram::setKernels (this=0x5555626f5480, binary=0x7ffff3547000, binSize=<optimized out>, fdesc=<optimized out>, foffset=<optimized out>, uri=...) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/device/rocm/rocprogram.cpp:328
#13 0x00007fffe897e8aa in amd::device::Program::loadLC (this=this@entry=0x5555626f5480) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/device/devprogram.cpp:1877
#14 0x00007fffe897e8e3 in amd::device::Program::load (this=this@entry=0x5555626f5480) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/device/devprogram.cpp:1888
#15 0x00007fffe89adb23 in amd::Program::load (this=0x5555562155b0, devices=...) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/platform/program.cpp:614
#16 0x00007fffe86b3fe8 in hip::FatBinaryInfo::BuildProgram (this=<optimized out>, device_id=device_id@entry=0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_fatbin.cpp:722
#17 0x00007fffe86ba5fb in hip::Function::getStatFunc (this=0x5555560970f0, hfunc=hfunc@entry=0x7fffffff7540, deviceId=deviceId@entry=0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_global.cpp:165
#18 0x00007fffe865ae9c in hip::StatCO::getStatFunc (this=0x555555b042e8, hfunc=hfunc@entry=0x7fffffff7540, hostFunction=hostFunction@entry=0x7fffedaffc20 <void mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int)>, deviceId=deviceId@entry=0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_code_object.cpp:1389
#19 0x00007fffe88a131d in hip::PlatformState::getStatFunc (this=<optimized out>, hfunc=hfunc@entry=0x7fffffff7540, hostFunction=hostFunction@entry=0x7fffedaffc20 <void mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int)>, deviceId=deviceId@entry=0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_platform.cpp:931
--Type <RET> for more, q to quit, c to continue without paging--thread apply 1 bt full
#20 0x00007fffe88a1410 in hip::ihipLaunchKernel (hostFunction=0x7fffedaffc20 <void mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int)>, gridDim=..., blockDim=..., args=0x7fffffff7cf0, sharedMemBytes=30336, stream=<optimized out>, startEvent=0x0, stopEvent=0x0, flags=0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_platform.cpp:625
#21 0x00007fffe88607ee in hip::hipLaunchKernel_common (hostFunction=<optimized out>, hostFunction@entry=0x7fffedaffc20 <void mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int)>, gridDim=..., blockDim=..., args=<optimized out>, args@entry=0x7fffffff7cf0, sharedMemBytes=<optimized out>, stream=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_module.cpp:738
#22 0x00007fffe88725e1 in hip::hipLaunchKernel (hostFunction=0x7fffedaffc20 <void mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int)>, gridDim=..., blockDim=..., args=<optimized out>, sharedMemBytes=<optimized out>, stream=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/hipamd/src/hip_module.cpp:745
#23 0x00007fffeda7f53c in void __device_stub__mul_mat_q<(ggml_type)3, 64, 8, false>(char const*, char const*, float*, float*, int, int, int, int, int, int, int) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#24 0x00007fffeda79297 in void launch_mul_mat_q<(ggml_type)3, 64>(ggml_backend_cuda_context&, mmq_args const&, ihipStream_t*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#25 0x00007fffeda82d40 in void mul_mat_q_case<(ggml_type)3>(ggml_backend_cuda_context&, mmq_args const&, ihipStream_t*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#26 0x00007fffed8f03d4 in ggml_cuda_op_mul_mat_q(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*, char const*, float const*, char const*, float*, long, long, long, long, ihipStream_t*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#27 0x00007fffed8e5669 in ggml_cuda_op_mul_mat(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*, void (*)(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*, char const*, float const*, char const*, float*, long, long, long, long, ihipStream_t*), void (*)(float const*, void*, ggml_type, long, long, long, long, long, long, long, long, ihipStream_t*)) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#28 0x00007fffed8e1aa2 in ggml_cuda_mul_mat(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#29 0x00007fffed8e1388 in ggml_cuda_compute_forward(ggml_backend_cuda_context&, ggml_tensor*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#30 0x00007fffed8e0f5b in evaluate_and_capture_cuda_graph(ggml_backend_cuda_context*, ggml_cgraph*, bool&, bool&, bool&) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#31 0x00007fffed8e0b6c in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/pl752/llama.cpp/build/bin/libggml-hip.so
#32 0x00007ffff7731e9c in ggml_backend_graph_compute_async () from /home/pl752/llama.cpp/build/bin/libggml-base.so
#33 0x00007ffff7735fd4 in ggml_backend_sched_compute_splits(ggml_backend_sched*) () from /home/pl752/llama.cpp/build/bin/libggml-base.so
#34 0x00007ffff7736c57 in ggml_backend_sched_graph_compute_async () from /home/pl752/llama.cpp/build/bin/libggml-base.so
#35 0x00007ffff7b5ce17 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/pl752/llama.cpp/build/bin/libllama.so
#36 0x00007ffff7b5b3a4 in llama_context::decode(llama_batch&) () from /home/pl752/llama.cpp/build/bin/libllama.so
#37 0x00007ffff7b6074e in llama_decode () from /home/pl752/llama.cpp/build/bin/libllama.so
#38 0x000055555565606f in server_context::update_slots() ()
#39 0x00005555555fcdc7 in main::{lambda()#4}::operator()() const ()
#40 0x000055555560b384 in void std::__invoke_impl<void, main::{lambda()#4}&>(std::__invoke_other, main::{lambda()#4}&) ()
#41 0x0000555555609254 in std::enable_if<is_invocable_r_v<void, main::{lambda()#4}&>, void>::type std::__invoke_r<void, main::{lambda()#4}&>(main::{lambda()#4}&) ()
#42 0x0000555555605483 in std::_Function_handler<void (), main::{lambda()#4}>::_M_invoke(std::_Any_data const&) ()
#43 0x000055555565bb08 in std::function<void ()>::operator()() const ()
#44 0x0000555555648c41 in server_queue::start_loop() ()
#45 0x00005555555ff634 in main ()

Thread 29 bt full

(gdb) thread apply 29 bt full

Thread 29 (Thread 0x7ffd737ff6c0 (LWP 21830) "llama-server"):
#0  __GI___ioctl (fd=fd@entry=3, request=request@entry=3222817548) at ../sysdeps/unix/sysv/linux/ioctl.c:36
        args = {{gp_offset = 16, fp_offset = 32765, overflow_arg_area = 0x7ffd737f5bc0, reg_save_area = 0x7ffd737f5b80}}
        arg = <optimized out>
        r = -512
#1  0x00007fffd6349400 in hsakmt_ioctl (fd=3, request=request@entry=3222817548, arg=arg@entry=0x7ffd737f5c30) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/libhsakmt.c:13
        ret = <optimized out>
#2  0x00007fffd6341f1f in hsaKmtWaitOnMultipleEvents_Ext (event_age=0x7ffd6c0019d0, Milliseconds=4294967294, WaitOnAll=false, NumEvents=<optimized out>, Events=0x7ffd6c001920) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:416
        result = <optimized out>
        event_data = 0x7ffd6c001350
        args = {events_ptr = 140726415397712, num_events = 2, wait_for_all = 0, timeout = 2147463933, wait_result = 2}
        out = <optimized out>
        result = <optimized out>
        event_data = <optimized out>
        args = <optimized out>
        out = <optimized out>
        i = <optimized out>
        i = <optimized out>
        i = <optimized out>
#3  hsaKmtWaitOnMultipleEvents_Ext (Events=0x7ffd6c001920, NumEvents=2, WaitOnAll=<optimized out>, Milliseconds=4294967294, event_age=0x7ffd6c0019d0) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:385
        result = <optimized out>
        event_data = <optimized out>
        args = <optimized out>
        out = <optimized out>
        i = <optimized out>
        i = <optimized out>
        i = <optimized out>
#4  0x00007fffd62a175c in operator() (__closure=<synthetic pointer>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/runtime.cpp:1601
        wait_ms = 4294967294
        end = <optimized out>
        hsa_events = <optimized out>
        unique_evts = <optimized out>
        event_age = <optimized out>
        hsa_events = <optimized out>
        unique_evts = <optimized out>
        event_age = <optimized out>
        wait_ms = <optimized out>
        end = <optimized out>
#5  rocr::core::Runtime::AsyncEventsLoop (_eventsInfo=0x5555560bd8b8) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime--Type <RET> for more, q to quit, c to continue without paging--c
/hsa-runtime/core/runtime/runtime.cpp:1691
        interrupt_wait = <optimized out>
        finish = false
        polling = false
        init_age = false
        value = {<std::_Vector_base<long, std::allocator<long> >> = {_M_impl = {<std::allocator<long>> = {<std::__new_allocator<long>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<long, std::allocator<long> >::_Vector_impl_data> = {_M_start = 0x55555e7111c0, _M_finish = 0x55555e7111c8, _M_end_of_storage = 0x55555e7111c8}, <No data fields>}}, <No data fields>}
        index = <optimized out>
        wait_any = <optimized out>
        functions = {<std::_Vector_base<std::pair<void (*)(void*), void*>, std::allocator<std::pair<void (*)(void*), void*> > >> = {_M_impl = {<std::allocator<std::pair<void (*)(void*), void*> >> = {<std::__new_allocator<std::pair<void (*)(void*), void*> >> = {<No data fields>}, <No data fields>}, <std::_Vector_base<std::pair<void (*)(void*), void*>, std::allocator<std::pair<void (*)(void*), void*> > >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, <No data fields>}}, <No data fields>}
        eventsInfo = 0x5555560bd8b8
        async_events_control_ = @0x5555560bd8b8: {wake = {handle = 140736783711104}, async_events_thread_ = 0x5555665840c0, lock = {lock_ = {<std::__atomic_base<int>> = {static _S_alignment = 4, _M_i = 0}, <No data fields>}, sem_ = 0x5555560b6370, maxSpinIterPause = 55, maxSpinIterYield = 55}, exit = false}
        async_events_ = @0x5555560bd8e8: {signal_ = {<std::_Vector_base<hsa_signal_s, std::allocator<hsa_signal_s> >> = {_M_impl = {<std::allocator<hsa_signal_s>> = {<std::__new_allocator<hsa_signal_s>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<hsa_signal_s, std::allocator<hsa_signal_s> >::_Vector_impl_data> = {_M_start = 0x7ffd6c001c80, _M_finish = 0x7ffd6c001d08, _M_end_of_storage = 0x7ffd6c001d80}, <No data fields>}}, <No data fields>}, cond_ = {<std::_Vector_base<hsa_signal_condition_t, std::allocator<hsa_signal_condition_t> >> = {_M_impl = {<std::allocator<hsa_signal_condition_t>> = {<std::__new_allocator<hsa_signal_condition_t>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<hsa_signal_condition_t, std::allocator<hsa_signal_condition_t> >::_Vector_impl_data> = {_M_start = 0x7ffd6c0013c0, _M_finish = 0x7ffd6c001404, _M_end_of_storage = 0x7ffd6c001440}, <No data fields>}}, <No data fields>}, value_ = {<std::_Vector_base<long, std::allocator<long> >> = {_M_impl = {<std::allocator<long>> = {<std::__new_allocator<long>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<long, std::allocator<long> >::_Vector_impl_data> = {_M_start = 0x7ffd6c001d90, _M_finish = 0x7ffd6c001e18, _M_end_of_storage = 0x7ffd6c001e90}, <No data fields>}}, <No data fields>}, handler_ = {<std::_Vector_base<bool (*)(long, void*), std::allocator<bool (*)(long, void*)> >> = {_M_impl = {<std::allocator<bool (*)(long, void*)>> = {<std::__new_allocator<bool (*)(long, void*)>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<bool (*)(long, void*), std::allocator<bool (*)(long, void*)> >::_Vector_impl_data> = {_M_start = 0x7ffd6c001ea0, _M_finish = 0x7ffd6c001f28, _M_end_of_storage = 0x7ffd6c001fa0}, <No data fields>}}, <No data fields>}, hsa_events_ = {<std::_Vector_base<_HsaEvent*, std::allocator<_HsaEvent*> >> = {_M_impl = {<std::allocator<_HsaEvent*>> = {<std::__new_allocator<_HsaEvent*>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<_HsaEvent*, std::allocator<_HsaEvent*> >::_Vector_impl_data> = {_M_start = 0x7ffd6c001920, _M_finish = 0x7ffd6c0019c0, _M_end_of_storage = 0x7ffd6c0019c0}, <No data fields>}}, <No data fields>}, age_ = {<std::_Vector_base<unsigned long, std::allocator<unsigned long> >> = {_M_impl = {<std::allocator<unsigned long>> = {<std::__new_allocator<unsigned long>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<unsigned long, std::allocator<unsigned long> >::_Vector_impl_data> = {_M_start = 0x7ffd6c0019d0, _M_finish = 0x7ffd6c001a70, _M_end_of_storage = 0x7ffd6c001a70}, <No data fields>}}, <No data fields>}, arg_ = {<std::_Vector_base<void*, std::allocator<void*> >> = {_M_impl = {<std::allocator<void*>> = {<std::__new_allocator<void*>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<void*, std::allocator<void*> >::_Vector_impl_data> = {_M_start = 0x7ffd6c001fb0, _M_finish = 0x7ffd6c002038, _M_end_of_storage = 0x7ffd6c0020b0}, <No data fields>}}, <No data fields>}}
        new_async_events_ = @0x5555560bd990: {signal_ = {<std::_Vector_base<hsa_signal_s, std::allocator<hsa_signal_s> >> = {_M_impl = {<std::allocator<hsa_signal_s>> = {<std::__new_allocator<hsa_signal_s>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<hsa_signal_s, std::allocator<hsa_signal_s> >::_Vector_impl_data> = {_M_start = 0x555566584480, _M_finish = 0x555566584480, _M_end_of_storage = 0x555566584490}, <No data fields>}}, <No data fields>}, cond_ = {<std::_Vector_base<hsa_signal_condition_t, std::allocator<hsa_signal_condition_t> >> = {_M_impl = {<std::allocator<hsa_signal_condition_t>> = {<std::__new_allocator<hsa_signal_condition_t>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<hsa_signal_condition_t, std::allocator<hsa_signal_condition_t> >::_Vector_impl_data> = {_M_start = 0x5555665841e0, _M_finish = 0x5555665841e0, _M_end_of_storage = 0x5555665841e8}, <No data fields>}}, <No data fields>}, value_ = {<std::_Vector_base<long, std::allocator<long> >> = {_M_impl = {<std::allocator<long>> = {<std::__new_allocator<long>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<long, std::allocator<long> >::_Vector_impl_data> = {_M_start = 0x555566584120, _M_finish = 0x555566584120, _M_end_of_storage = 0x555566584130}, <No data fields>}}, <No data fields>}, handler_ = {<std::_Vector_base<bool (*)(long, void*), std::allocator<bool (*)(long, void*)> >> = {_M_impl = {<std::allocator<bool (*)(long, void*)>> = {<std::__new_allocator<bool (*)(long, void*)>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<bool (*)(long, void*), std::allocator<bool (*)(long, void*)> >::_Vector_impl_data> = {_M_start = 0x555566584420, _M_finish = 0x555566584420, _M_end_of_storage = 0x555566584430}, <No data fields>}}, <No data fields>}, hsa_events_ = {<std::_Vector_base<_HsaEvent*, std::allocator<_HsaEvent*> >> = {_M_impl = {<std::allocator<_HsaEvent*>> = {<std::__new_allocator<_HsaEvent*>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<_HsaEvent*, std::allocator<_HsaEvent*> >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, <No data fields>}}, <No data fields>}, age_ = {<std::_Vector_base<unsigned long, std::allocator<unsigned long> >> = {_M_impl = {<std::allocator<unsigned long>> = {<std::__new_allocator<unsigned long>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<unsigned long, std::allocator<unsigned long> >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, <No data fields>}}, <No data fields>}, arg_ = {<std::_Vector_base<void*, std::allocator<void*> >> = {_M_impl = {<std::allocator<void*>> = {<std::__new_allocator<void*>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<void*, std::allocator<void*> >::_Vector_impl_data> = {_M_start = 0x555566584440, _M_finish = 0x555566584440, _M_end_of_storage = 0x555566584450}, <No data fields>}}, <No data fields>}}
        hsa_events = @0x5555560bd948: {<std::_Vector_base<_HsaEvent*, std::allocator<_HsaEvent*> >> = {_M_impl = {<std::allocator<_HsaEvent*>> = {<std::__new_allocator<_HsaEvent*>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<_HsaEvent*, std::allocator<_HsaEvent*> >::_Vector_impl_data> = {_M_start = 0x7ffd6c001920, _M_finish = 0x7ffd6c0019c0, _M_end_of_storage = 0x7ffd6c0019c0}, <No data fields>}}, <No data fields>}
        event_age = @0x5555560bd960: {<std::_Vector_base<unsigned long, std::allocator<unsigned long> >> = {_M_impl = {<std::allocator<unsigned long>> = {<std::__new_allocator<unsigned long>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<unsigned long, std::allocator<unsigned long> >::_Vector_impl_data> = {_M_start = 0x7ffd6c0019d0, _M_finish = 0x7ffd6c001a70, _M_end_of_storage = 0x7ffd6c001a70}, <No data fields>}}, <No data fields>}
        unique_evts = <optimized out>
        hsa_signals = 0x7ffd6c001c80
        processEvent = {__async_events_ = @0x5555560bd8e8, __hsa_signals = <synthetic pointer><error reading variable: Cannot access memory at address 0x0>}
        checkCondition = <optimized out>
        PrepareInterrupt = {__hsa_signals = @0x7ffd737f5d10, __unique_evts = @0x7ffd737f5d10, __hsa_events = @0x5555560bd948, __event_age = @0x5555560bd960}
        WaitForInterrupt = {__hsa_events = @0x5555560bd948, __unique_evts = <synthetic pointer><error reading variable: Cannot access memory at address 0x0>, __event_age = @0x5555560bd960}
#6  0x00007fffd6239241 in rocr::os::ThreadTrampoline (arg=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/util/lnx/os_linux.cpp:86
        ar = <optimized out>
        CallMe = <optimized out>
        Data = <optimized out>
#7  0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140726541219520, 2996067661364555957, 4294967295, -5344, 0, 140737488318464, 2996067661284864181, 2995214015928561845}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#8  0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
No locals.

Thread 31 bt full

(gdb) thread apply 31 bt full

Thread 31 (Thread 0x7fffd65bf6c0 (LWP 21836) "llama-server"):
#0  __GI___ioctl (fd=fd@entry=3, request=request@entry=3222817548) at ../sysdeps/unix/sysv/linux/ioctl.c:36
        args = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7fffd65b5aa0, reg_save_area = 0x7fffd65b5a60}}
        arg = <optimized out>
        r = -512
#1  0x00007fffd6349400 in hsakmt_ioctl (fd=3, request=request@entry=3222817548, arg=arg@entry=0x7fffd65b5b10) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/libhsakmt.c:13
        ret = <optimized out>
#2  0x00007fffd6341f1f in hsaKmtWaitOnMultipleEvents_Ext (event_age=0x7fffd65b5c18, Milliseconds=15, WaitOnAll=true, NumEvents=<optimized out>, Events=0x7fffd65b5bc8) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:416
        result = <optimized out>
        event_data = 0x7ffc38000e30
        args = {events_ptr = 140721248013872, num_events = 1, wait_for_all = 1, timeout = 11, wait_result = 2}
        out = <optimized out>
        result = <optimized out>
        event_data = <optimized out>
        args = <optimized out>
        out = <optimized out>
        i = <optimized out>
        i = <optimized out>
        i = <optimized out>
#3  hsaKmtWaitOnMultipleEvents_Ext (Events=0x7fffd65b5bc8, NumEvents=1, WaitOnAll=<optimized out>, Milliseconds=15, event_age=0x7fffd65b5c18) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:385
        result = <optimized out>
        event_data = <optimized out>
        args = <optimized out>
        out = <optimized out>
        i = <optimized out>
        i = <optimized out>
        i = <optimized out>
#4  0x00007fffd63426db in hsaKmtWaitOnEvent_Ext (Event=<optimized out>, Milliseconds=<optimized out>, event_age=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/libhsakmt/src/events.c:231
No locals.
#5  0x00007fffd627cd3b in rocr::core::InterruptSignal::WaitRelaxed (this=0x5555668958a0, condition=HSA_SIGNAL_CONDITION_NE, compare_value=1, timeout=<optimized out>, wait_hint=HSA_WAIT_STATE_BLOCKED) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/interrupt_signal.cpp:197
        value = 1
        now = <optimized out>
        remaining_ms = <optimized out>
        wait_ms = <optimized out>
        scopeGuardLambda3 = {__this = 0x5555668958a0}
        scopeGuard4 = {release_ = {__this = 0x5555668958a0}, dismiss_ = false}
        prior = <optimized out>
        scopeGuardLambda5 = {__this = 0x5555668958a0}
--Type <RET> for more, q to quit, c to continue without paging--c
        scopeGuard6 = {release_ = {__this = 0x5555668958a0}, dismiss_ = false}
        event_age = 2
        start_time = <optimized out>
        fast_timeout = {__r = 16000000000}
        kMaxElapsed = {__r = 200000000}
        signal_abort_timeout = <optimized out>
#6  0x00007fffd627ca3e in rocr::core::InterruptSignal::WaitAcquire (this=<optimized out>, condition=<optimized out>, compare_value=<optimized out>, timeout=<optimized out>, wait_hint=<optimized out>) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/interrupt_signal.cpp:205
        ret = <optimized out>
#7  0x00007fffd6270c81 in rocr::HSA::hsa_signal_wait_scacquire (hsa_signal=..., condition=HSA_SIGNAL_CONDITION_NE, compare_value=1, timeout_hint=16000000, wait_state_hint=HSA_WAIT_STATE_BLOCKED) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/ROCR-Runtime/runtime/hsa-runtime/core/runtime/hsa.cpp:1249
        signal = <optimized out>
#8  0x00007fffe89f7d55 in amd::HostcallListener::consumePackets (this=0x555567386c30) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/device/devhostcall.cpp:307
        new_value = <optimized out>
        timeout = 16000000
        signal_value = 1
#9  0x00007fffe8657412 in amd::Thread::main (this=this@entry=0x555567386cc8) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/thread/thread.cpp:93
No locals.
#10 0x00007fffe899bfa6 in amd::Thread::entry (thread=0x555567386cc8) at /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/clr/rocclr/os/os_posix.cpp:351
        set = {__val = {128, 0, 0, 11664321608736097024, 0, 311296, 140736789430272, 140736789699952, 140736789700032, 140737338001089, 2322277832189568800, 2334097595223798896, 3611951134054380659, 909326385, 0, 11664321608736097024}}
#11 0x00007ffff709caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736789739200, 2995282850580492469, 140736789739200, -5344, 0, 140737488321360, 2995282850500800693, 2995214015928561845}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#12 0x00007ffff7129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
No locals.

Apr 25 '25 09:04 pl752

This issue was closed because it has been inactive for 14 days since being marked as stale.

Jun 09 '25 01:06 github-actions[bot]