uvloop python: src/unix/core.c:930: uv__io_stop: Assertion `loop->watchers[w->fd] == w' failed.

python: src/unix/core.c:930: uv__io_stop: Assertion `loop->watchers[w->fd] == w' failed.

Open alviezhang opened this issue 3 years ago • 3 comments

uvloop version: 0.15.2, 0.15.3, 0.16.0.dev0
Python version: 3.8.10
Platform: Linux/Ubuntu 20.04
Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: Yes
Does uvloop behave differently from vanilla asyncio? How?: ~I don't know how to start starlette use vanilla asyncio.~ I try use hypercorn to run vanilla asyncio asgi server, which has no coredump happened. And I use hypercorn with uvloop worker, the coredump happens again.

I use starlette, aiohttp HTTP client and uvicorn in production, and I found server automatically restarted when server is getting traffic increased.

Our starlette server requests third party website and return their responses. I use aiohttp as http client, sometime it uses HTTP proxy. I found that if the proxy is disabled, I won't get coredump. And I tried to provide an simple code to reproduce the core dump and I failed, can I do something to debug what is going wrong to get rid of the coredump?

Here is the error log of uvicorn server:

python: src/unix/core.c:930: uv__io_stop: Assertion `loop->watchers[w->fd] == w' failed.

[1]    107718 abort (core dumped)  PYTHONASYNCIODEBUG=1 ./run_server.sh

And I compiled uvloop with debug symbol, here are two of coredump stack

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f4579796859 in __GI_abort () at abort.c:79
#2  0x00007f4579796729 in __assert_fail_base (fmt=0x7f457992c588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7f4570b4e551 "loop->watchers[w->fd] == w",
    file=0x7f4570b4e380 "src/unix/core.c", line=930, function=<optimized out>) at assert.c:92
#3  0x00007f45797a7f36 in __GI___assert_fail (assertion=0x7f4570b4e551 "loop->watchers[w->fd] == w", file=0x7f4570b4e380 "src/unix/core.c", line=930,
    function=0x7f4570b4e668 <__PRETTY_FUNCTION__.9575> "uv__io_stop") at assert.c:101
#4  0x00007f4570b1cf42 in uv__io_stop (loop=0x2e3b340, w=0xba6d0b8, events=1) at src/unix/core.c:930
#5  0x00007f4570b284f3 in uv_read_stop (stream=0xba6d030) at src/unix/stream.c:1595
#6  0x00007f45709ce30f in __pyx_f_6uvloop_4loop_8UVStream__stop_reading (__pyx_v_self=0xc3cdf40) at uvloop/loop.c:95318
#7  0x00007f45709bdba3 in __pyx_pf_6uvloop_4loop_15UVBaseTransport_6_force_close (__pyx_v_self=0xc3cdf40, __pyx_v_exc=None) at uvloop/loop.c:90709
#8  0x00007f45709bd6af in __pyx_pw_6uvloop_4loop_15UVBaseTransport_7_force_close (__pyx_v_self=<uvloop.loop.TCPTransport at remote 0xc3cdf40>, __pyx_v_exc=None)
    at uvloop/loop.c:90620
#9  0x00007f4570b07fe8 in __Pyx_PyObject_CallMethO (func=<built-in method _force_close of uvloop.loop.TCPTransport object at remote 0xc3cdf40>, arg=None) at uvloop/loop.c:179161
#10 0x00007f4570b08266 in __Pyx_PyObject_CallOneArg (func=<built-in method _force_close of uvloop.loop.TCPTransport object at remote 0xc3cdf40>, arg=None)
    at uvloop/loop.c:179214
#11 0x00007f4570a83afa in __pyx_f_6uvloop_4loop_11SSLProtocol__abort (__pyx_v_self=0x7f4564935040, __pyx_v_exc=None) at uvloop/loop.c:141862
#12 0x00007f4570a7dc85 in __pyx_f_6uvloop_4loop_11SSLProtocol__start_shutdown (__pyx_v_self=0x7f4564935040, __pyx_optional_args=0x7ffc3255bb30) at uvloop/loop.c:140300
#13 0x00007f4570a5fd4b in __pyx_pf_6uvloop_4loop_21_SSLProtocolTransport_10close (__pyx_v_self=0x7f4555f900c0) at uvloop/loop.c:132514
#14 0x00007f4570a5f91c in __pyx_pw_6uvloop_4loop_21_SSLProtocolTransport_11close (__pyx_v_self=<uvloop.loop._SSLProtocolTransport at remote 0x7f4555f900c0>, unused=0x0)
    at uvloop/loop.c:132456
#15 0x00007f4570b07fe8 in __Pyx_PyObject_CallMethO (func=<built-in method close of uvloop.loop._SSLProtocolTransport object at remote 0x7f4555f900c0>, arg=0x0)
    at uvloop/loop.c:179161
#16 0x00007f4570b0812e in __Pyx_PyObject_CallNoArg (func=<built-in method close of uvloop.loop._SSLProtocolTransport object at remote 0x7f4555f900c0>) at uvloop/loop.c:179187
#17 0x00007f45708f6553 in __pyx_gb_6uvloop_4loop_4Loop_60generator4 (__pyx_generator=0x7f454eaeb400, __pyx_tstate=0x1e486e0, __pyx_sent_value=0x0) at uvloop/loop.c:39370
#18 0x00007f4570b0e084 in __Pyx_Coroutine_SendEx (self=0x7f454eaeb400, value=0x0, closing=0) at uvloop/loop.c:182028
#19 0x00007f4570b0e264 in __Pyx_Coroutine_FinishDelegation (gen=0x7f454eaeb400) at uvloop/loop.c:182095
#20 0x00007f4570b0ed26 in __Pyx__Coroutine_Throw (self=<iterable_coroutine at remote 0x7f454eaeb400>, typ=<CancelledError at remote 0x7f454df3ac40>, val=0x0, tb=0x0,
    args=(<CancelledError at remote 0x7f454df3ac40>,), close_on_genexit=1) at uvloop/loop.c:182334
#21 0x00007f4570b0ee26 in __Pyx_Coroutine_Throw (self=<iterable_coroutine at remote 0x7f454eaeb400>, args=(<CancelledError at remote 0x7f454df3ac40>,)) at uvloop/loop.c:182348
#22 0x00007f4570b0fc5e in __Pyx_CoroutineAwait_Throw (self=0x7f454f817880, args=(<CancelledError at remote 0x7f454df3ac40>,)) at uvloop/loop.c:182778
#23 0x00000000005f2d6a in cfunction_call_varargs (kwargs=<optimized out>, args=<optimized out>,
    func=<built-in method throw of coroutine_wrapper object at remote 0x7f454f817880>) at ../Include/internal/pycore_pyerrors.h:13
#24 PyCFunction_Call (func=<built-in method throw of coroutine_wrapper object at remote 0x7f454f817880>, args=<optimized out>, kwargs=<optimized out>) at ../Objects/call.c:773
#25 0x00000000005f3010 in _PyObject_MakeTpCall (callable=<built-in method throw of coroutine_wrapper object at remote 0x7f454f817880>, args=0x7ffc3255c7e0,
    nargs=<optimized out>, keywords=0x0) at ../Objects/call.c:159
#26 0x00000000005f360d in _PyObject_Vectorcall (kwnames=0x0, nargsf=1, args=0x7ffc3255c7e0,
    callable=<built-in method throw of coroutine_wrapper object at remote 0x7f454f817880>) at ../Include/cpython/abstract.h:125
#27 _PyObject_Vectorcall (kwnames=0x0, nargsf=1, args=0x7ffc3255c7e0, callable=<built-in method throw of coroutine_wrapper object at remote 0x7f454f817880>)
    at ../Include/cpython/abstract.h:115
#28 _PyObject_FastCall (nargs=1, args=0x7ffc3255c7e0, func=<built-in method throw of coroutine_wrapper object at remote 0x7f454f817880>) at ../Include/cpython/abstract.h:147
#29 object_vacall (base=<optimized out>, callable=<built-in method throw of coroutine_wrapper object at remote 0x7f454f817880>, vargs=<optimized out>) at ../Objects/call.c:1187
#30 0x00000000005f3795 in PyObject_CallFunctionObjArgs (callable=<optimized out>) at ../Objects/call.c:1260
#31 0x0000000000613539 in _gen_throw (gen=0x7f455741e340, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:442
#32 0x00000000006135ae in _gen_throw (gen=0x7f454cdb5040, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#33 0x00000000006135ae in _gen_throw (gen=0x7f454cdb5a40, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#34 0x00000000006135ae in _gen_throw (gen=0x7f454cdb56c0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#35 0x00000000006135ae in _gen_throw (gen=0x7f4565a70bc0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#36 0x00000000006135ae in _gen_throw (gen=0x7f4565a70b40, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#37 0x00000000006135ae in _gen_throw (gen=0x7f4565a70cc0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#38 0x00000000006135ae in _gen_throw (gen=0x7f4565a70f40, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#39 0x00000000006135ae in _gen_throw (gen=0x7f4565a70640, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#40 0x00000000006135ae in _gen_throw (gen=0x7f4565a706c0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#41 0x00000000006135ae in _gen_throw (gen=0x7f4565a707c0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#42 0x00000000006135ae in _gen_throw (gen=0x7f4565a70840, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#43 0x00000000006135ae in _gen_throw (gen=0x7f4565a70340, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#44 0x00000000006135ae in _gen_throw (gen=0x7f454f815940, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#45 0x00000000006135ae in _gen_throw (gen=0x7f454f8151c0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#46 0x00000000006135ae in _gen_throw (gen=0x7f454f815240, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#47 0x00000000006135ae in _gen_throw (gen=0x7f454f815040, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#48 0x00000000006135ae in _gen_throw (gen=0x7f454f815540, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#49 0x00000000006135ae in _gen_throw (gen=0x7f454f815e40, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#50 0x00000000006135ae in _gen_throw (gen=0x7f454f8155c0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#51 0x00000000006135ae in _gen_throw (gen=0x7f454f815ac0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#52 0x00000000006135ae in _gen_throw (gen=0x7f454f815b40, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#53 0x00000000006135ae in _gen_throw (gen=0x7f454f8158c0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#54 0x00000000006135ae in _gen_throw (gen=0x7f454f815cc0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#55 0x00000000006135ae in _gen_throw (gen=0x7f454f815740, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#56 0x00000000006135ae in _gen_throw (gen=0x7f454f815840, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#57 0x00000000006135ae in _gen_throw (gen=0x7f454f815dc0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#58 0x00000000006135ae in _gen_throw (gen=0x7f456c1439c0, close_on_genexit=1, typ=<optimized out>, val=<optimized out>, tb=<optimized out>) at ../Objects/genobject.c:427
#59 0x000000000061362d in gen_throw (gen=0x7f456c1439c0, args=<optimized out>) at ../Objects/genobject.c:536
#60 0x0000000000505a1d in method_vectorcall_VARARGS (func=<optimized out>, args=0x7ffc3255d280, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/descrobject.c:300
#61 0x00000000005f34b1 in _PyObject_Vectorcall (kwnames=0x0, nargsf=2, args=0x7ffc3255d280, callable=<method_descriptor at remote 0x7f45792afb30>)
    at ../Include/cpython/abstract.h:127
#62 _PyObject_FastCall (nargs=2, args=0x7ffc3255d280, func=<method_descriptor at remote 0x7f45792afb30>) at ../Include/cpython/abstract.h:147
#63 object_vacall (base=<optimized out>, callable=<method_descriptor at remote 0x7f45792afb30>, vargs=<optimized out>) at ../Objects/call.c:1187
#64 0x00000000005f3938 in _PyObject_CallMethodIdObjArgs (obj=<coroutine at remote 0x7f456c1439c0>, name=name@entry=0x7f4579174280 <PyId_throw>) at ../Objects/call.c:1245
#65 0x00007f4579169c95 in task_step_impl (exc=<CancelledError at remote 0x7f454df3ac40>, task=0x7f45566e3720) at ./Modules/_asynciomodule.c:2649
#66 task_step (task=0x7f45566e3720, exc=<optimized out>) at ./Modules/_asynciomodule.c:2934
#67 0x00007f457916a027 in task_wakeup (o=<optimized out>, task=0x7f45566e3720) at ./Modules/_asynciomodule.c:2993
#68 TaskWakeupMethWrapper_call (o=0x7f4554d9b850, args=<optimized out>, kwds=<optimized out>) at ./Modules/_asynciomodule.c:1791
#69 0x00007f4570b07d78 in __Pyx_PyObject_Call (func=<TaskWakeupMethWrapper at remote 0x7f4554d9b850>, arg=(<_asyncio.Future at remote 0x7f455741ec40>,), kw=0x0)
    at uvloop/loop.c:179099
#70 0x00007f4570952410 in __pyx_f_6uvloop_4loop_6Handle__run (__pyx_v_self=0x7f4566a399d0) at uvloop/loop.c:61745
#71 0x00007f4570887bb8 in __pyx_f_6uvloop_4loop_4Loop__on_idle (__pyx_v_self=0x2e88de0) at uvloop/loop.c:14363
#72 0x00007f4570952595 in __pyx_f_6uvloop_4loop_6Handle__run (__pyx_v_self=0x7f4570bc0b80) at uvloop/loop.c:61772
#73 0x00007f457099d5ae in __pyx_f_6uvloop_4loop_cb_idle_callback (__pyx_v_handle=0x2c18a00) at uvloop/loop.c:81786
#74 0x00007f4570b1fcfb in uv__run_idle (loop=0x2e3b340) at src/unix/loop-watcher.c:68
#75 0x00007f4570b1bdd4 in uv_run (loop=0x2e3b340, mode=UV_RUN_DEFAULT) at src/unix/core.c:378
#76 0x00007f457088b150 in __pyx_f_6uvloop_4loop_4Loop___run (__pyx_v_self=0x2e88de0, __pyx_v_mode=UV_RUN_DEFAULT) at uvloop/loop.c:15256
#77 0x00007f457088cdc7 in __pyx_f_6uvloop_4loop_4Loop__run (__pyx_v_self=0x2e88de0, __pyx_v_mode=UV_RUN_DEFAULT) at uvloop/loop.c:15677
#78 0x00007f45708c8fbe in __pyx_pf_6uvloop_4loop_4Loop_24run_forever (__pyx_v_self=0x2e88de0) at uvloop/loop.c:29012
#79 0x00007f45708c7e8f in __pyx_pw_6uvloop_4loop_4Loop_25run_forever (__pyx_v_self=<Loop at remote 0x2e88de0>, unused=0x0) at uvloop/loop.c:28820
#80 0x00007f4570b07fe8 in __Pyx_PyObject_CallMethO (func=<built-in method run_forever of Loop object at remote 0x2e88de0>, arg=0x0) at uvloop/loop.c:179161
#81 0x00007f4570b0812e in __Pyx_PyObject_CallNoArg (func=<built-in method run_forever of Loop object at remote 0x2e88de0>) at uvloop/loop.c:179187
#82 0x00007f45708cf35a in __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (__pyx_v_self=0x2e88de0, __pyx_v_future=<_asyncio.Task at remote 0x7f4570bc5400>)
    at uvloop/loop.c:30693
#83 0x00007f45708cd604 in __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (__pyx_v_self=<Loop at remote 0x2e88de0>, __pyx_v_future=<coroutine at remote 0x7f4570ba7cc0>)
    at uvloop/loop.c:30272
#84 0x0000000000504d56 in method_vectorcall_O (func=<method_descriptor at remote 0x7f457080ed10>, args=0x7f4570f22e08, nargsf=<optimized out>, kwnames=<optimized out>)
    at ../Objects/descrobject.c:416
#85 0x000000000056acb6 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f4570f22e08, callable=<method_descriptor at remote 0x7f457080ed10>)
    at ../Include/cpython/abstract.h:127
#86 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x1e486e0) at ../Python/ceval.c:4963
#87 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3486
#89 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>,
    kwargs=0x7f4570bbb1f0, kwcount=<optimized out>, kwstep=1, defs=0x7f457859d148, defcount=1, kwdefs=0x0, closure=0x0, name='run', qualname='Server.run')
    at ../Python/ceval.c:4298
#90 0x00000000005f5b33 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x7f4570bbb1e8, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:436
#91 0x000000000056acb6 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f4570bbb1e8, callable=<function at remote 0x7f4578f6fa60>)
    at ../Include/cpython/abstract.h:127
#92 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x1e486e0) at ../Python/ceval.c:4963
#93 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3486
...
#97 0x000000000056bc9b in _PyObject_Vectorcall (kwnames=('host', 'port'), nargsf=<optimized out>, args=<optimized out>, callable=<function at remote 0x7f457859c9d0>)
    at ../Include/cpython/abstract.h:127
#98 call_function (kwnames=('host', 'port'), oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at ../Python/ceval.c:4963
#99 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3515
#100 0x0000000000568d9a in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7f457923c840, for file crawler/main.py, line 7, in <module> ()) at ../Python/ceval.c:741
#101 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>,
    kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at ../Python/ceval.c:4298
#102 0x000000000068cdc7 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=<optimized out>,
    globals=<optimized out>, _co=<optimized out>) at ../Python/ceval.c:4327
#103 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at ../Python/ceval.c:718
...
#107 0x000000000067e627 in pyrun_simple_file (flags=0x7ffc3255f368, closeit=1, filename='crawler/main.py', fp=0x1e48d70) at ../Python/pythonrun.c:439
#108 PyRun_SimpleFileExFlags (fp=0x1e48d70, filename=<optimized out>, closeit=1, flags=0x7ffc3255f368) at ../Python/pythonrun.c:472
#109 0x00000000006b6e62 in pymain_run_file (cf=0x7ffc3255f368, config=0x1e49180) at ../Modules/main.c:385
#110 pymain_run_python (exitcode=0x7ffc3255f360) at ../Modules/main.c:610
#111 Py_RunMain () at ../Modules/main.c:689
#112 0x00000000006b71ed in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:743
#113 0x00007f45797980b3 in __libc_start_main (main=0x4ef190 <main>, argc=2, argv=0x7ffc3255f548, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7ffc3255f538) at ../csu/libc-start.c:308
#114 0x00000000005f96de in _start () at ../Include/internal/pycore_pyerrors.h:13

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f74c0f6c859 in __GI_abort () at abort.c:79
#2  0x00007f74c0f6c729 in __assert_fail_base (fmt=0x7f74c1102588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=0x7f74b8324551 "loop->watchers[w->fd] == w", file=0x7f74b8324380 "src/unix/core.c", line=930, function=<optimized out>)
    at assert.c:92
#3  0x00007f74c0f7df36 in __GI___assert_fail (assertion=0x7f74b8324551 "loop->watchers[w->fd] == w",
    file=0x7f74b8324380 "src/unix/core.c", line=930, function=0x7f74b8324668 <__PRETTY_FUNCTION__.9575> "uv__io_stop") at assert.c:101
#4  0x00007f74b82f2f42 in uv__io_stop (loop=0x2fb4ee0, w=0x15eb7ea8, events=8199) at src/unix/core.c:930
#5  0x00007f74b82f3047 in uv__io_close (loop=0x2fb4ee0, w=0x15eb7ea8) at src/unix/core.c:943
#6  0x00007f74b82fe5ee in uv__stream_close (handle=0x15eb7e20) at src/unix/stream.c:1658
#7  0x00007f74b82ff61c in uv__tcp_close (handle=0x15eb7e20) at src/unix/tcp.c:460
#8  0x00007f74b82f1772 in uv_close (handle=0x15eb7e20, close_cb=0x7f74b816afa2 <__pyx_f_6uvloop_4loop___uv_close_handle_cb>)
    at src/unix/core.c:124
#9  0x00007f74b816310b in __pyx_f_6uvloop_4loop_8UVHandle__close (__pyx_v_self=0x18987580) at uvloop/loop.c:77393
#10 0x00007f74b81665b5 in __pyx_f_6uvloop_4loop_14UVSocketHandle__close (__pyx_v_self=0x18987580) at uvloop/loop.c:78354
#11 0x00007f74b81aed95 in __pyx_f_6uvloop_4loop_8UVStream__close (__pyx_v_self=0x18987580) at uvloop/loop.c:97796
#12 0x00007f74b818f816 in __pyx_f_6uvloop_4loop_15UVBaseTransport__call_connection_lost (__pyx_v_self=0x18987580,
    __pyx_v_exc=OSError(9, 'Bad file descriptor')) at uvloop/loop.c:89384
#13 0x00007f74b8128743 in __pyx_f_6uvloop_4loop_6Handle__run (__pyx_v_self=0x7f749df0e4c0) at uvloop/loop.c:61799
#14 0x00007f74b805dbb8 in __pyx_f_6uvloop_4loop_4Loop__on_idle (__pyx_v_self=0x3002890) at uvloop/loop.c:14363
#15 0x00007f74b8128595 in __pyx_f_6uvloop_4loop_6Handle__run (__pyx_v_self=0x7f74b8398af0) at uvloop/loop.c:61772
#16 0x00007f74b81735ae in __pyx_f_6uvloop_4loop_cb_idle_callback (__pyx_v_handle=0x2d8ed80) at uvloop/loop.c:81786
#17 0x00007f74b82f5cfb in uv__run_idle (loop=0x2fb4ee0) at src/unix/loop-watcher.c:68
#18 0x00007f74b82f1dd4 in uv_run (loop=0x2fb4ee0, mode=UV_RUN_DEFAULT) at src/unix/core.c:378
#19 0x00007f74b8061150 in __pyx_f_6uvloop_4loop_4Loop___run (__pyx_v_self=0x3002890, __pyx_v_mode=UV_RUN_DEFAULT) at uvloop/loop.c:15256
#20 0x00007f74b8062dc7 in __pyx_f_6uvloop_4loop_4Loop__run (__pyx_v_self=0x3002890, __pyx_v_mode=UV_RUN_DEFAULT) at uvloop/loop.c:15677
#21 0x00007f74b809efbe in __pyx_pf_6uvloop_4loop_4Loop_24run_forever (__pyx_v_self=0x3002890) at uvloop/loop.c:29012
#22 0x00007f74b809de8f in __pyx_pw_6uvloop_4loop_4Loop_25run_forever (__pyx_v_self=<Loop at remote 0x3002890>, unused=0x0)
    at uvloop/loop.c:28820
#23 0x00007f74b82ddfe8 in __Pyx_PyObject_CallMethO (func=<built-in method run_forever of Loop object at remote 0x3002890>, arg=0x0)
    at uvloop/loop.c:179161
#24 0x00007f74b82de12e in __Pyx_PyObject_CallNoArg (func=<built-in method run_forever of Loop object at remote 0x3002890>)
    at uvloop/loop.c:179187
#25 0x00007f74b80a535a in __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (__pyx_v_self=0x3002890,
    __pyx_v_future=<_asyncio.Task at remote 0x7f74b839c400>) at uvloop/loop.c:30693
#26 0x00007f74b80a3604 in __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (__pyx_v_self=<Loop at remote 0x3002890>, __pyx_v_future=<coroutine at remote 0x7f74b837ddc0>) at uvloop/loop.c:30272
#27 0x0000000000504d56 in method_vectorcall_O (func=<method_descriptor at remote 0x7f74b7fe5d10>, args=0x7f74b86fde08,
    nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/descrobject.c:416
#28 0x000000000056acb6 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f74b86fde08,
    callable=<method_descriptor at remote 0x7f74b7fe5d10>) at ../Include/cpython/abstract.h:127
#29 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x1fc06e0) at ../Python/ceval.c:4963
#30 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3486

#31 0x0000000000568d9a in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f74b86fdc80, for file /home/alvie/code/aftership/xxx/.venv/lib/python3.8/site-packages/uvicorn/server.py, line 49, in run (self=<Server(config=<Config(app=<Starlette(_debug=True, state=<State(_state={'engine': <CrawlerEngine(_connector_lock=<Lock(_waiters=None, _locked=False, _loop=<Loop at remote 0x3002890>) at remote 0x7f74b7b0a880>, _loader=...) at ../Python/ceval.c:741
#32 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>,
    argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x7f74b83923d0, kwcount=<optimized out>, kwstep=1, defs=0x7f74bfd74148,
    defcount=1, kwdefs=0x0, closure=0x0, name='run', qualname='Server.run') at ../Python/ceval.c:4298
#33 0x00000000005f5b33 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x7f74b83923c8, nargsf=<optimized out>,
    kwnames=<optimized out>) at ../Objects/call.c:436
#34 0x000000000056acb6 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f74b83923c8,
    callable=<function at remote 0x7f74c0745a60>) at ../Include/cpython/abstract.h:127
#35 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x1fc06e0) at ../Python/ceval.c:4963
#36 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3486
#37 0x0000000000568d9a in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f74b8392220, for file /home/alvie/code/xxx/.venv/lib/python3.8/site-packages/uvicorn/main.py, line 642, in run (app=<Starlette(_debug=True, state=<State(_state={'engine': <CrawlerEngine(_connector_lock=<Lock(_waiters=None, _locked=False, _loop=<Loop at remote 0x3002890>) at remote 0x7f74b7b0a880>, _loader=...) at ../Python/ceval.c:741
#38 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>,
    argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x7f74c0a129b8, kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name='run', qualname='run') at ../Python/ceval.c:4298
#39 0x00000000005f5b33 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x7f74c0a129b0, nargsf=<optimized out>,
    kwnames=<optimized out>) at ../Objects/call.c:436
#40 0x000000000056bc9b in _PyObject_Vectorcall (kwnames=('host', 'port'), nargsf=<optimized out>, args=<optimized out>,
    callable=<function at remote 0x7f74bfd739d0>) at ../Include/cpython/abstract.h:127
#41 call_function (kwnames=('host', 'port'), oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>)
    at ../Python/ceval.c:4963
#42 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3515
#43 0x0000000000568d9a in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7f74c0a12840, for file crawler/main.py, line 7, in <module> ())
    at ../Python/ceval.c:741
#44 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>,
    argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0,
    closure=0x0, name=0x0, qualname=0x0) at ../Python/ceval.c:4298
#45 0x000000000068cdc7 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0,
    locals=<optimized out>, globals=<optimized out>, _co=<optimized out>) at ../Python/ceval.c:4327
#46 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at ../Python/ceval.c:718

#47 0x000000000067e161 in run_eval_code_obj (co=0x7f74c09d0660,
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='crawler/main.py') at remote 0x7f74c09bf220>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f74c0a240e0>, '__file__': 'crawler/main.py', '__cached__': None, 'uvicorn': <module at remote 0x7f74c087cf90>, ...,
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='crawler/main.py') at remote 0x7f74c09bf220>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f74c0a240e0>, '__file__': 'crawler/main.py', '__cached__': None, 'uvicorn': <module at remote 0x7f74c087cf90>, ...) at ../Python/pythonrun.c:1166
#48 0x000000000067e1df in run_mod (mod=<optimized out>, filename=<optimized out>,
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='crawler/main.py') at remote 0x7f74c09bf220>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f74c0a240e0>, '__file__': 'crawler/main.py', '__cached__': None, 'uvicorn': <module at remote 0x7f74c087cf90>, ..., flags=<optimized out>, arena=<optimized out>)
    at ../Python/pythonrun.c:1188
#49 0x000000000067e281 in pyrun_file (fp=fp@entry=0x1fc0d70, filename=filename@entry='crawler/main.py', start=start@entry=257,

    globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='crawler/main.py') at remote 0x7f74c09bf220>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f74c0a240e0>, '__file__': 'crawler/main.py', '__cached__': None, 'uvicorn': <module at remote 0x7f74c087cf90>, ... , closeit=closeit@entry=1, flags=0x7ffd969d35a8)
    at ../Python/pythonrun.c:1085
#50 0x000000000067e627 in pyrun_simple_file (flags=0x7ffd969d35a8, closeit=1, filename='crawler/main.py', fp=0x1fc0d70)
    at ../Python/pythonrun.c:439
#51 PyRun_SimpleFileExFlags (fp=0x1fc0d70, filename=<optimized out>, closeit=1, flags=0x7ffd969d35a8) at ../Python/pythonrun.c:472
#52 0x00000000006b6e62 in pymain_run_file (cf=0x7ffd969d35a8, config=0x1fc1180) at ../Modules/main.c:385
#53 pymain_run_python (exitcode=0x7ffd969d35a0) at ../Modules/main.c:610
#54 Py_RunMain () at ../Modules/main.c:689
#55 0x00000000006b71ed in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:743
#56 0x00007f74c0f6e0b3 in __libc_start_main (main=0x4ef190 <main>, argc=2, argv=0x7ffd969d3788, init=<optimized out>,
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd969d3778) at ../csu/libc-start.c:308
#57 0x00000000005f96de in _start () at ../Include/internal/pycore_pyerrors.h:13

Jul 20 '21 02:07 alviezhang

Exactly the same situation here! Aiohttp with proxy will produce the same error log and the coredump info.

Dec 13 '21 02:12 Jiramew

/cc @1st1, @fantix, @elprans. This issue leads to aborting of whole Python process.

Jul 08 '22 12:07 navytux

From the stack, it looks like it's either we used libuv in the wrong way, or we ran into some bug in libuv. Given that this one is very similar to #317, let's try to find a way to reproduce reliably first. What I tried:

docker run --rm -e TZ=UTC -p 3128:3128 ubuntu/squid
Install and run NGINX in the container
Run the following script:

import asyncio
import uvloop
import aiohttp
from asyncio import queues

uvloop.install()
C = 128


async def task(i, q):
    async with aiohttp.ClientSession() as session:
        while True:
            if await q.get():
                break
            async with session.get(
                "http://localhost", proxy="http://localhost:3128"
            ) as resp:
                print(i, resp.status)


async def main():
    q = queues.Queue()
    ts = []
    for i in range(C):
        ts.append(asyncio.create_task(task(i, q)))
    for i in range(65536):
        q.put_nowait(False)
    for i in range(C):
        q.put_nowait(True)
    await asyncio.wait(ts)


asyncio.run(main())

Unfortunately for me it completes without crashing 🥀

Sep 09 '22 19:09 fantix

uvloop uvloop copied to clipboard

python: src/unix/core.c:930: uv__io_stop: Assertion `loop->watchers[w->fd] == w' failed.

uvloop
uvloop copied to clipboard