uvloop
uvloop copied to clipboard
MagicStack
Segmentation fault
- uvloop version: 0.16.0
- Python version: Python 3.7.10
- Platform: Amazon Linux release 2 (Karoo)
It happens couple of times a day sometimes. And than does not happen for weeks:
Thread 2 "python3.7" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe71b3700 (LWP 31714)]
0x0000000000000000 in ?? ()
Missing separate debuginfos, use: debuginfo-install python3-3.7.10-1.amzn2.0.1.x86_64
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x00007fffe97b2a5b in uv__read (stream=stream@entry=0x7fffe01bbe40) at src/unix/stream.c:1211
#2 0x00007fffe97b2f58 in uv__stream_io (loop=<optimized out>, w=0x7fffe01bbec8, events=1) at src/unix/stream.c:1316
#3 0x00007fffe97b9e75 in uv.io_poll () at src/unix/epoll.c:374
#4 0x00007fffe97ac684 in uv_run (loop=0x9982e0, mode=mode@entry=UV_RUN_DEFAULT) at src/unix/core.c:389
#5 0x00007fffe975e1f0 in __pyx_f_6uvloop_4loop_4Loop___run (__pyx_v_self=0xa19e70, __pyx_v_mode=UV_RUN_DEFAULT) at uvloop/loop.c:14642
#6 0x00007fffe976548d in __pyx_f_6uvloop_4loop_4Loop__run (__pyx_v_self=0xa19e70, __pyx_v_mode=UV_RUN_DEFAULT) at uvloop/loop.c:15030
#7 0x00007fffe96ebbc8 in __pyx_pf_6uvloop_4loop_4Loop_24run_forever (__pyx_v_self=0xa19e70) at uvloop/loop.c:27698
#8 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (__pyx_v_self=0xa19e70, unused=<optimized out>) at uvloop/loop.c:27519
#9 0x00007ffff79d9db0 in _PyMethodDef_RawFastCallKeywords () from /lib64/libpython3.7m.so.1.0
#10 0x00007ffff79f37ed in _PyMethodDescr_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#11 0x00007ffff7a37cfb in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#12 0x00007ffff79cce66 in _PyFunction_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#13 0x00007ffff7a3326f in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#14 0x00007ffff79cce66 in _PyFunction_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#15 0x00007ffff7a3326f in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#16 0x00007ffff7995757 in _PyFunction_FastCallDict () from /lib64/libpython3.7m.so.1.0
#17 0x00007ffff79a2711 in _PyObject_Call_Prepend () from /lib64/libpython3.7m.so.1.0
#18 0x00007ffff7995baa in PyObject_Call () from /lib64/libpython3.7m.so.1.0
#19 0x00007ffff7aa0b43 in t_bootstrap () from /lib64/libpython3.7m.so.1.0
#20 0x00007ffff7a4a147 in pythread_wrapper () from /lib64/libpython3.7m.so.1.0
#21 0x00007ffff741744b in start_thread () from /lib64/libpthread.so.0
#22 0x00007ffff6a0b40f in clone () from /lib64/libc.so.6
My app is using uvloop + aiohttp to continiously poll certain websites with high frequency. There are couple of threads which a sending tasks to a single thread where uvloop loop runs.
Both processes where this reproduces run at -17 and -20 niceness. There are some other processes with niceness around -10,-12 where it never reproduced. So maybe somehow related.
The error crashes the Python process. Also here's some additional info gathered after crash:
Id Target Id Frame
1 Thread 0x7ffff7fe3740 (LWP 31681) "python3.7" 0x00007ffff741fa46 in do_futex_wait.constprop ()
from /lib64/libpthread.so.0
* 2 Thread 0x7fffe71b3700 (LWP 31714) "python3.7" 0x0000000000000000 in ?? ()
3 Thread 0x7fffdf312700 (LWP 31735) "python3.7" 0x00007ffff741fa46 in do_futex_wait.constprop ()
from /lib64/libpthread.so.0
4 Thread 0x7fffde911700 (LWP 31736) "python3.7" 0x00007ffff6a01519 in poll () from /lib64/libc.so.6
5 Thread 0x7fffdd58f700 (LWP 31738) "python3.7" 0x00007ffff6a01519 in poll () from /lib64/libc.so.6
6 Thread 0x7fffcf54a700 (LWP 31742) "python3.7" 0x00007ffff6a01519 in poll () from /lib64/libc.so.6
7 Thread 0x7fffceb49700 (LWP 31743) "python3.7" 0x00007ffff741fc10 in do_futex_wait ()
from /lib64/libpthread.so.0
8 Thread 0x7fffce148700 (LWP 31744) "python3.7" 0x00007ffff741d377 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
9 Thread 0x7fffcd747700 (LWP 31745) "python3.7" 0x00007ffff741d377 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
10 Thread 0x7fffccd46700 (LWP 31746) "python3.7" 0x00007ffff741d377 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
11 Thread 0x7fffc7fff700 (LWP 31747) "python3.7" 0x00007ffff741d377 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
12 Thread 0x7fffc75fe700 (LWP 31748) "python3.7" 0x00007ffff6a032bc in select () from /lib64/libc.so.6
13 Thread 0x7fffc6bfd700 (LWP 31749) "python3.7" 0x00007ffff6a032bc in select () from /lib64/libc.so.6```
Yes it reproduces with 0.17.0 as well.
However I think I found the reason for the issue. It was caused by using this function to cleanup aiohttp connections:
def remove_connections(removed_hosts):
# Remove not-used connections to prevent GC cycle dependencies
for conn_key in list(session.connector._conns.keys()):
if conn_key.host in removed_hosts:
try:
response_handler_tuples = session.connector._conns[conn_key]
except KeyError:
continue
for response_handler_tuple in response_handler_tuples:
response_handler_tuple[0].close()
response_handler_tuple[0]._parser = None
try:
del session.connector._conns[conn_key]
except KeyError:
pass
Maybe it removed a connection which was being polled by uvloop or something. But once I removed this code the issue stopped reproducing.