lmdeploy
lmdeploy copied to clipboard
Published
20 hours ago •
InternLM
InternLM
[Bug] 访问一段时间后服务卡死/无响应
Checklist
- [X] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
Describe the bug
单卡A100上跑qwenvl-chat模型，使用32k的session-len跑一段时间请求后，服务失去响应。
Reproduction
1、lmdeploy serve api_server Qwen-VL-Chat --server-port 80 --session-len 32768 2、使用client跑一段时间后，服务失去响应，client端如果使用同步，没有设置超时时间的话，client一直没有响应返回
Environment
单卡A100
lmdeploy check_env
/usr/local/lib/python3.10/dist-packages/gradio_client/documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Parallel'>: No known documentation group for module 'gradio.mix'
warnings.warn(f"Could not get documentation group for {cls}: {exc}")
/usr/local/lib/python3.10/dist-packages/gradio_client/documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Series'>: No known documentation group for module 'gradio.mix'
warnings.warn(f"Could not get documentation group for {cls}: {exc}")
sys.platform: linux
Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0: NVIDIA A100-SXM4-80GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.3, V12.3.107
GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.2.2+cu121
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.3.2 (Git Hash 2dc95a2ad0841e29db8b22fbccaf3e5da7992b01)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 12.1
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
- CuDNN 8.9.2
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits-Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.2.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,
TorchVision: 0.17.2+cu121
LMDeploy: 0.4.2+
transformers: 4.40.2
gradio: 3.40.1
fastapi: 0.110.2
pydantic: 2.7.1
triton: 2.2.0
Error traceback
#0 0x00007f8a5a52fb68 in __GI_epoll_pwait (epfd=epfd@entry=42, events=events@entry=0x7fffb8baaba0, maxevents=maxevents@entry=1024,
timeout=timeout@entry=100, set=set@entry=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:40
#1 0x00007f896cd2f23d in uv__io_poll (loop=0x557fda8f8ee0, timeout=100) at src/unix/linux.c:1359
#2 0x00007f896cd21ea6 in uv_run (loop=0x557fda8f8ee0, mode=UV_RUN_DEFAULT) at src/unix/core.c:447
#3 0x00007f896ccada83 in __pyx_f_6uvloop_4loop_4Loop___run (__pyx_v_self=0x557f04badca0, __pyx_v_mode=UV_RUN_DEFAULT)
at uvloop/loop.c:15092
#4 0x00007f896ccb6b16 in __pyx_f_6uvloop_4loop_4Loop__run (__pyx_v_self=0x557f04badca0, __pyx_v_mode=UV_RUN_DEFAULT)
at uvloop/loop.c:15471
#5 0x00007f896cc7c3c3 in __pyx_pf_6uvloop_4loop_4Loop_24run_forever (__pyx_v_self=0x557f04badca0) at uvloop/loop.c:28166
#6 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (__pyx_v_self=0x557f04badca0, unused=<optimized out>) at uvloop/loop.c:27987
#7 0x00007f896cc48355 in __Pyx_PyObject_CallMethO (arg=0x0, func=0x7f899d838860) at uvloop/loop.c:171467
#8 __Pyx_PyObject_CallNoArg (func=0x7f899d838860) at uvloop/loop.c:40421
#9 0x00007f896cce128d in __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (__pyx_v_future=0x7f896ce38e10,
__pyx_v_self=<optimized out>) at uvloop/loop.c:30035
#10 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (__pyx_v_self=<optimized out>, __pyx_v_future=<optimized out>)
at uvloop/loop.c:29633
#11 0x0000557ebff992ca in ?? ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7bcfa in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#16 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#17 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#18 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#19 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#20 0x0000557ebff9c492 in PyObject_Call ()
#21 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#22 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#23 0x0000557ebff7bcfa in _PyEval_EvalFrameDefault ()
#24 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#25 0x0000557ebff7626d in _PyEval_EvalFrameDefault ()
#26 0x0000557ebff729c6 in ?? ()
#27 0x0000557ec0068256 in PyEval_EvalCode ()
#28 0x0000557ec0093108 in ?? ()
#29 0x0000557ec008c9cb in ?? ()
#30 0x0000557ec0092e55 in ?? ()
#31 0x0000557ec0092338 in _PyRun_SimpleFileObject ()
#32 0x0000557ec0091f83 in _PyRun_AnyFileObject ()
#33 0x0000557ec0084a5e in Py_RunMain ()
#34 0x0000557ec005b02d in Py_BytesMain ()
#35 0x00007f8a5a433d90 in __libc_start_call_main (main=main@entry=0x557ec005aff0, argc=argc@entry=9, argv=argv@entry=0x7fffb8baef98)
at ../sysdeps/nptl/libc_start_call_main.h:58
#36 0x00007f8a5a433e40 in __libc_start_main_impl (main=0x557ec005aff0, argc=9, argv=0x7fffb8baef98, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffb8baef88) at ../csu/libc-start.c:392
#37 0x0000557ec005af25 in _start ()
Thread 31 (Thread 0x7f83fe7fc640 (LWP 248689) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f83a8478890) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x7f83a8478890) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f83a8478890, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x7f83a8478890, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x7f83a8478890, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x0000557ebffa4461 in ?? ()
#7 0x0000557ebff8e8b7 in ?? ()
#8 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#16 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#17 0x0000557ebff9ba51 in ?? ()
#18 0x0000557ec00c4f3a in ?? ()
#19 0x0000557ec00b9ef8 in ?? ()
#20 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#21 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 30 (Thread 0x7f83feffd640 (LWP 238135) "lmdeploy"):
#0 0x00007f8a58218b31 in ?? () from /usr/local/lib/python3.10/dist-packages/torch/lib/libgomp-a34b3233.so.1
#1 0x00007f8a582163e0 in ?? () from /usr/local/lib/python3.10/dist-packages/torch/lib/libgomp-a34b3233.so.1
#2 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#3 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 29 (Thread 0x7f83ff7fe640 (LWP 238134) "lmdeploy"):
#0 0x00007f8a58218b31 in ?? () from /usr/local/lib/python3.10/dist-packages/torch/lib/libgomp-a34b3233.so.1
#1 0x00007f8a582163e0 in ?? () from /usr/local/lib/python3.10/dist-packages/torch/lib/libgomp-a34b3233.so.1
#2 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#3 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 28 (Thread 0x7f83fffff640 (LWP 238133) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129ddf in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 27 (Thread 0x7f841cff9640 (LWP 238132) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129ddf in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 26 (Thread 0x7f841d7fa640 (LWP 238131) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129ddf in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 25 (Thread 0x7f841dffb640 (LWP 238130) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129ddf in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 24 (Thread 0x7f841e7fc640 (LWP 238129) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129f72 in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 23 (Thread 0x7f841effd640 (LWP 238128) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129f72 in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 22 (Thread 0x7f841f7fe640 (LWP 238127) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129f72 in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 21 (Thread 0x7f841ffff640 (LWP 238126) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129ddf in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 20 (Thread 0x7f86d6ffd640 (LWP 238125) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129f72 in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 19 (Thread 0x7f86d77fe640 (LWP 238124) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129ddf in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 18 (Thread 0x7f86d7fff640 (LWP 238123) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129ddf in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 17 (Thread 0x7f87f2ffd640 (LWP 238122) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129f72 in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 16 (Thread 0x7f87f37fe640 (LWP 238121) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x557fda93c1b0) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557fda93c1b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a6bdf in do_futex_wait (sem=sem@entry=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a6c78 in __new_sem_wait_slow64 (sem=0x557fda93c1b0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c5d0 in PyThread_acquire_lock_timed ()
#6 0x00007f8a5a129f72 in ?? () from /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
#7 0x0000557ec003f094 in ?? ()
#8 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff9ba51 in ?? ()
#16 0x0000557ec00c4f3a in ?? ()
#17 0x0000557ec00b9ef8 in ?? ()
#18 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 15 (Thread 0x7f89408f9640 (LWP 238107) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f896cd8e728 <cond+40>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f896cd8e728 <cond+40>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f896cd8e728 <cond+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a49da41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f896cd8e6c0 <mutex>, cond=0x7f896cd8e700 <cond>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x7f896cd8e700 <cond>, mutex=0x7f896cd8e6c0 <mutex>) at ./nptl/pthread_cond_wait.c:627
#5 0x00007f896cd2acc9 in uv_cond_wait (cond=<optimized out>, mutex=<optimized out>) at src/unix/thread.c:793
#6 0x00007f896cd1d659 in worker (arg=0x0) at src/threadpool.c:76
#7 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 14 (Thread 0x7f89410fa640 (LWP 238106) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f896cd8e728 <cond+40>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f896cd8e728 <cond+40>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f896cd8e728 <cond+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a49da41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f896cd8e6c0 <mutex>, cond=0x7f896cd8e700 <cond>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x7f896cd8e700 <cond>, mutex=0x7f896cd8e6c0 <mutex>) at ./nptl/pthread_cond_wait.c:627
#5 0x00007f896cd2acc9 in uv_cond_wait (cond=<optimized out>, mutex=<optimized out>) at src/unix/thread.c:793
#6 0x00007f896cd1d659 in worker (arg=0x0) at src/threadpool.c:76
#7 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 13 (Thread 0x7f89418fb640 (LWP 238105) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f896cd8e728 <cond+40>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f896cd8e728 <cond+40>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f896cd8e728 <cond+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a49da41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f896cd8e6c0 <mutex>, cond=0x7f896cd8e700 <cond>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x7f896cd8e700 <cond>, mutex=0x7f896cd8e6c0 <mutex>) at ./nptl/pthread_cond_wait.c:627
#5 0x00007f896cd2acc9 in uv_cond_wait (cond=<optimized out>, mutex=<optimized out>) at src/unix/thread.c:793
#6 0x00007f896cd1d659 in worker (arg=0x0) at src/threadpool.c:76
#7 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 12 (Thread 0x7f89430fe640 (LWP 238104) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f896cd8e72c <cond+44>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f896cd8e72c <cond+44>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f896cd8e72c <cond+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a49da41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f896cd8e6c0 <mutex>, cond=0x7f896cd8e700 <cond>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x7f896cd8e700 <cond>, mutex=0x7f896cd8e6c0 <mutex>) at ./nptl/pthread_cond_wait.c:627
#5 0x00007f896cd2acc9 in uv_cond_wait (cond=<optimized out>, mutex=<optimized out>) at src/unix/thread.c:793
#6 0x00007f896cd1d659 in worker (arg=0x0) at src/threadpool.c:76
#7 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 11 (Thread 0x7f89420fc640 (LWP 237976) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f8439346f14) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f8439346f14) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f8439346f14, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a49da41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f8439346ec0, cond=0x7f8439346ee8) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x7f8439346ee8, mutex=0x7f8439346ec0) at ./nptl/pthread_cond_wait.c:627
#5 0x00007f8a58080747 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007f899280361e in std::condition_variable::wait<turbomind::LlamaBatch<__nv_bfloat16>::OutputThreadEntry()::{lambda()#1}>(std::unique_lock<std::mutex>&, turbomind::LlamaBatch<__nv_bfloat16>::OutputThreadEntry()::{lambda()#1}) (__p=..., __lock=..., this=0x7f8439346ee8) at /opt/rh/devtoolset-9/root/usr/include/c++/9/condition_variable:101
#7 turbomind::LlamaBatch<__nv_bfloat16>::OutputThreadEntry (this=0x7f8439346ac0) at /lmdeploy/src/turbomind/models/llama/LlamaBatch.cc:1501
#8 0x00007f8a580b0253 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 10 (Thread 0x7f89428fd640 (LWP 237975) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x557ed528ba34) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x557ed528ba34) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x557ed528ba34, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a49da41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x557ed528b9e0, cond=0x557ed528ba08) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x557ed528ba08, mutex=0x557ed528b9e0) at ./nptl/pthread_cond_wait.c:627
#5 0x00007f8a58080747 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007f89928027f0 in std::condition_variable::wait<turbomind::RequestQueue::dequeue(std::vector<std::shared_ptr<turbomind::Request>, std::allocator<std::shared_ptr<turbomind::Request> > >&, std::vector<std::shared_ptr<turbomind::Request>, std::allocator<std::shared_ptr<turbomind::Request> > >&, unsigned int, bool, bool&)::{lambda()#1}>(std::unique_lock<std::mutex>&, turbomind::RequestQueue::dequeue(std::vector<std::shared_ptr<turbomind::Request>, std::allocator<std::shared_ptr<turbomind::Request> > >&, std::vector<std::shared_ptr<turbomind::Request>, std::allocator<std::shared_ptr<turbomind::Request> > >&, unsigned int, bool, bool&)::{lambda()#1}) (__p=..., __lock=..., this=0x557ed528ba08) at /opt/rh/devtoolset-9/root/usr/include/c++/9/condition_variable:101
#7 turbomind::RequestQueue::dequeue (this=this@entry=0x557ed528b940, stop_requests=std::vector of length 0, capacity 0, infer_requests=std::vector of length 0, capacity 1, max_infer_count=max_infer_count@entry=128, blocking=blocking@entry=true, abort=@0x557ed528ba50: false) at /lmdeploy/src/turbomind/models/llama/Request.h:77
#8 0x00007f89928321e0 in turbomind::LlamaBatch<__nv_bfloat16>::InternalThreadEntry (this=<optimized out>, device_id=<optimized out>) at /lmdeploy/src/turbomind/models/llama/LlamaBatch.cc:1409
#9 0x00007f8a580b0253 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 9 (Thread 0x7f89438ff640 (LWP 237961) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x7f89438fe330, op=137, expected=0, futex_word=0x7f8938263030) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x7f89438fe330, clockid=0, expected=0, futex_word=0x7f8938263030) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f8938263030, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f89438fe330, private=<optimized out>) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a4a5ed0 in do_futex_wait (sem=sem@entry=0x7f8938263030, clockid=clockid@entry=1, abstime=abstime@entry=0x7f89438fe330) at ./nptl/sem_waitcommon.c:111
#4 0x00007f8a5a4a5f7e in __new_sem_wait_slow64 (sem=0x7f8938263030, clockid=1, abstime=0x7f89438fe330) at ./nptl/sem_waitcommon.c:183
#5 0x0000557ebff4c62b in PyThread_acquire_lock_timed ()
#6 0x0000557ebffa4461 in ?? ()
#7 0x0000557ebff8e8b7 in ?? ()
#8 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#9 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#10 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#11 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#16 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#17 0x0000557ebff9ba51 in ?? ()
#18 0x0000557ec00c4f3a in ?? ()
#19 0x0000557ec00b9ef8 in ?? ()
#20 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#21 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 8 (Thread 0x7f896e4dd640 (LWP 237946) "lmdeploy"):
#0 0x00007f8a5a52fe2e in epoll_wait (epfd=38, events=0x7f89439d9090, maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x0000557ec00e43d2 in ?? ()
#2 0x0000557ebffae55e in ?? ()
#3 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#4 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#5 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#6 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#7 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#8 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#9 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#10 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#11 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#12 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#13 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#14 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#15 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#16 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#17 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#18 0x0000557ebff9ba51 in ?? ()
#19 0x0000557ec00c4f3a in ?? ()
#20 0x0000557ec00b9ef8 in ?? ()
#21 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#22 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 7 (Thread 0x7f896feff640 (LWP 237923) "cuda-EvtHandlr"):
#0 0x00007f8a5a522bcf in __GI___poll (fds=0x7f8964000c20, nfds=11, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f89adc33b3f in ?? () from /usr/local/cuda/compat/lib.real/libcuda.so.1
#2 0x00007f89adcf66cf in ?? () from /usr/local/cuda/compat/lib.real/libcuda.so.1
#3 0x00007f89adc2fd2f in ?? () from /usr/local/cuda/compat/lib.real/libcuda.so.1
#4 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#5 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 6 (Thread 0x7f89808fe640 (LWP 237922) "lmdeploy"):
#0 0x00007f8a58218b31 in ?? () from /usr/local/lib/python3.10/dist-packages/torch/lib/libgomp-a34b3233.so.1
#1 0x00007f8a582163e0 in ?? () from /usr/local/lib/python3.10/dist-packages/torch/lib/libgomp-a34b3233.so.1
#2 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#3 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 5 (Thread 0x7f89810ff640 (LWP 237921) "lmdeploy"):
#0 0x00007f8a58218b31 in ?? () from /usr/local/lib/python3.10/dist-packages/torch/lib/libgomp-a34b3233.so.1
#1 0x00007f8a582163e0 in ?? () from /usr/local/lib/python3.10/dist-packages/torch/lib/libgomp-a34b3233.so.1
#2 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#3 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 4 (Thread 0x7f898efe2640 (LWP 237920) "cuda00001400006"):
#0 0x00007f8a5a522bcf in __GI___poll (fds=0x557ec7d95f80, nfds=3, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f89adc33b3f in ?? () from /usr/local/cuda/compat/lib.real/libcuda.so.1
#2 0x00007f89adcf66cf in ?? () from /usr/local/cuda/compat/lib.real/libcuda.so.1
#3 0x00007f89adc2fd2f in ?? () from /usr/local/cuda/compat/lib.real/libcuda.so.1
#4 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#5 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 3 (Thread 0x7f89a63fe640 (LWP 237918) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f89a947de60 <thread_status+224>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f89a947de60 <thread_status+224>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f89a947de60 <thread_status+224>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a49da41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f89a947de10 <thread_status+144>, cond=0x7f89a947de38 <thread_status+184>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x7f89a947de38 <thread_status+184>, mutex=0x7f89a947de10 <thread_status+144>) at ./nptl/pthread_cond_wait.c:627
#5 0x00007f89a78036fb in blas_thread_server () from /usr/local/lib/python3.10/dist-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
#6 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 2 (Thread 0x7f89a6bff640 (LWP 237917) "lmdeploy"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f89a947dde0 <thread_status+96>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f89a947dde0 <thread_status+96>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f89a947dde0 <thread_status+96>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f8a5a49da41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f89a947dd90 <thread_status+16>, cond=0x7f89a947ddb8 <thread_status+56>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x7f89a947ddb8 <thread_status+56>, mutex=0x7f89a947dd90 <thread_status+16>) at ./nptl/pthread_cond_wait.c:627
#5 0x00007f89a78036fb in blas_thread_server () from /usr/local/lib/python3.10/dist-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
#6 0x00007f8a5a49eac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7 0x00007f8a5a52fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Thread 1 (Thread 0x7f8a5a405740 (LWP 237915) "lmdeploy"):
#0 0x00007f8a5a52fb68 in __GI_epoll_pwait (epfd=epfd@entry=42, events=events@entry=0x7fffb8baaba0, maxevents=maxevents@entry=1024, timeout=timeout@entry=100, set=set@entry=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:40
#1 0x00007f896cd2f23d in uv__io_poll (loop=0x557fda8f8ee0, timeout=100) at src/unix/linux.c:1359
#2 0x00007f896cd21ea6 in uv_run (loop=0x557fda8f8ee0, mode=UV_RUN_DEFAULT) at src/unix/core.c:447
#3 0x00007f896ccada83 in __pyx_f_6uvloop_4loop_4Loop___run (__pyx_v_self=0x557f04badca0, __pyx_v_mode=UV_RUN_DEFAULT) at uvloop/loop.c:15092
#4 0x00007f896ccb6b16 in __pyx_f_6uvloop_4loop_4Loop__run (__pyx_v_self=0x557f04badca0, __pyx_v_mode=UV_RUN_DEFAULT) at uvloop/loop.c:15471
#5 0x00007f896cc7c3c3 in __pyx_pf_6uvloop_4loop_4Loop_24run_forever (__pyx_v_self=0x557f04badca0) at uvloop/loop.c:28166
#6 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (__pyx_v_self=0x557f04badca0, unused=<optimized out>) at uvloop/loop.c:27987
#7 0x00007f896cc48355 in __Pyx_PyObject_CallMethO (arg=0x0, func=0x7f899d838860) at uvloop/loop.c:171467
#8 __Pyx_PyObject_CallNoArg (func=0x7f899d838860) at uvloop/loop.c:40421
#9 0x00007f896cce128d in __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (__pyx_v_future=0x7f896ce38e10, __pyx_v_self=<optimized out>) at uvloop/loop.c:30035
#10 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (__pyx_v_self=<optimized out>, __pyx_v_future=<optimized out>) at uvloop/loop.c:29633
#11 0x0000557ebff992ca in ?? ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7bcfa in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#16 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#17 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#18 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#19 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#20 0x0000557ebff9c492 in PyObject_Call ()
#21 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#22 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#23 0x0000557ebff7bcfa in _PyEval_EvalFrameDefault ()
#24 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#25 0x0000557ebff7626d in _PyEval_EvalFrameDefault ()
#26 0x0000557ebff729c6 in ?? ()
#27 0x0000557ec0068256 in PyEval_EvalCode ()
#28 0x0000557ec0093108 in ?? ()
#29 0x0000557ec008c9cb in ?? ()
#30 0x0000557ec0092e55 in ?? ()
#31 0x0000557ec0092338 in _PyRun_SimpleFileObject ()
#32 0x0000557ec0091f83 in _PyRun_AnyFileObject ()
#33 0x0000557ec0084a5e in Py_RunMain ()
#34 0x0000557ec005b02d in Py_BytesMain ()
#35 0x00007f8a5a433d90 in __libc_start_call_main (main=main@entry=0x557ec005aff0, argc=argc@entry=9, argv=argv@entry=0x7fffb8baef98) at ../sysdeps/nptl/libc_start_call_main.h:58
#36 0x00007f8a5a433e40 in __libc_start_main_impl (main=0x557ec005aff0, argc=9, argv=0x7fffb8baef98, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffb8baef88) at ../csu/libc-start.c:392
#37 0x0000557ec005af25 in _start ()
Program not restarted.
Continuing.
[New Thread 0x7f87f3fff640 (LWP 271082)]
[Thread 0x7f87f3fff640 (LWP 271082) exited]
Thread 1 "lmdeploy" received signal SIGINT, Interrupt.
0x00007f8a5a52fb68 in __GI_epoll_pwait (epfd=epfd@entry=42, events=events@entry=0x7fffb8baaba0, maxevents=maxevents@entry=1024, timeout=timeout@entry=39, set=set@entry=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:40
40 in ../sysdeps/unix/sysv/linux/epoll_pwait.c
#0 0x00007f8a5a52fb68 in __GI_epoll_pwait (epfd=epfd@entry=42, events=events@entry=0x7fffb8baaba0, maxevents=maxevents@entry=1024,
timeout=timeout@entry=39, set=set@entry=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:40
#1 0x00007f896cd2f23d in uv__io_poll (loop=0x557fda8f8ee0, timeout=39) at src/unix/linux.c:1359
#2 0x00007f896cd21ea6 in uv_run (loop=0x557fda8f8ee0, mode=UV_RUN_DEFAULT) at src/unix/core.c:447
#3 0x00007f896ccada83 in __pyx_f_6uvloop_4loop_4Loop___run (__pyx_v_self=0x557f04badca0, __pyx_v_mode=UV_RUN_DEFAULT)
at uvloop/loop.c:15092
#4 0x00007f896ccb6b16 in __pyx_f_6uvloop_4loop_4Loop__run (__pyx_v_self=0x557f04badca0, __pyx_v_mode=UV_RUN_DEFAULT)
at uvloop/loop.c:15471
#5 0x00007f896cc7c3c3 in __pyx_pf_6uvloop_4loop_4Loop_24run_forever (__pyx_v_self=0x557f04badca0) at uvloop/loop.c:28166
#6 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (__pyx_v_self=0x557f04badca0, unused=<optimized out>) at uvloop/loop.c:27987
#7 0x00007f896cc48355 in __Pyx_PyObject_CallMethO (arg=0x0, func=0x7f899d838860) at uvloop/loop.c:171467
#8 __Pyx_PyObject_CallNoArg (func=0x7f899d838860) at uvloop/loop.c:40421
#9 0x00007f896cce128d in __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (__pyx_v_future=0x7f896ce38e10,
__pyx_v_self=<optimized out>) at uvloop/loop.c:30035
#10 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (__pyx_v_self=<optimized out>, __pyx_v_future=<optimized out>)
at uvloop/loop.c:29633
#11 0x0000557ebff992ca in ?? ()
#12 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#13 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#14 0x0000557ebff7bcfa in _PyEval_EvalFrameDefault ()
#15 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#16 0x0000557ebff7645c in _PyEval_EvalFrameDefault ()
#17 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#18 0x0000557ebff7753c in _PyEval_EvalFrameDefault ()
#19 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#20 0x0000557ebff9c492 in PyObject_Call ()
#21 0x0000557ebff785d7 in _PyEval_EvalFrameDefault ()
#22 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#23 0x0000557ebff7bcfa in _PyEval_EvalFrameDefault ()
#24 0x0000557ebff8d9fc in _PyFunction_Vectorcall ()
#25 0x0000557ebff7626d in _PyEval_EvalFrameDefault ()
#26 0x0000557ebff729c6 in ?? ()
#27 0x0000557ec0068256 in PyEval_EvalCode ()
#28 0x0000557ec0093108 in ?? ()
#29 0x0000557ec008c9cb in ?? ()
#30 0x0000557ec0092e55 in ?? ()
#31 0x0000557ec0092338 in _PyRun_SimpleFileObject ()
#32 0x0000557ec0091f83 in _PyRun_AnyFileObject ()
#33 0x0000557ec0084a5e in Py_RunMain ()
#34 0x0000557ec005b02d in Py_BytesMain ()
#35 0x00007f8a5a433d90 in __libc_start_call_main (main=main@entry=0x557ec005aff0, argc=argc@entry=9, argv=argv@entry=0x7fffb8baef98)
at ../sysdeps/nptl/libc_start_call_main.h:58
#36 0x00007f8a5a433e40 in __libc_start_main_impl (main=0x557ec005aff0, argc=9, argv=0x7fffb8baef98, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffb8baef88) at ../csu/libc-start.c:392
#37 0x0000557ec005af25 in _start ()
Detaching from program: /usr/bin/python3.10, process 237915
