Bug report

This is a regression from the previous versions of Python 3.10 - 3.7.

python3.11: Objects/object.c:2276: _PyTrash_thread_destroy_chain: Assertion `tstate->trash_delete_nesting == 1' failed.

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ff69548ecb3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007ff69543e9c6 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ff6954287f4 in __GI_abort () at abort.c:79
#4  0x00007ff69542871b in __assert_fail_base (fmt=0x7ff6955bbda0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2276, 
    function=<optimized out>) at assert.c:92
#5  0x00007ff695437576 in __GI___assert_fail (assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2276, function=0x786cc0 <__PRETTY_FUNCTION__.3> "_PyTrash_thread_destroy_chain")
    at assert.c:101
#6  0x0000000000549867 in _PyTrash_thread_destroy_chain () at Objects/object.c:2276
#7  0x000000000054992b in _PyTrash_end (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2301
#8  0x000000000055c1eb in tupledealloc (op=0x7ff686941c20) at Objects/tupleobject.c:215
#9  0x0000000000549ba2 in _Py_Dealloc (
    op=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), 83), None), 55), None), 13), None), 42), None), 70), None), 7), None), 8), None), 51), None), None), ()), (...)), (...)), None), None), 15), None), 64), None), 94), None), 76), None), None), None), None), None), None), 91), None), 41), None), None), None), None), 31), None), 4), None), None), None), 78), None), 79), None), None), None), None), None), 92), None), 39), None), 23), None), 93), None), 68), None), 47), None), 74), None), 84), None), 72), None), 98), None), 61), None), 90), None), None), 99), 33)) at Objects/object.c:2389
#10 0x00007ff6877dbd1d in Py_DECREF (op=<optimized out>) at /home/arcivanov/.pyenv/versions/3.11.0rc2/include/python3.11/object.h:538
#11 __pyx_f_6gevent_17_gevent_cgreenlet_8Greenlet__Greenlet__free (__pyx_v_self=__pyx_v_self@entry=0x7ff686929310) at src/gevent/greenlet.c:14915
#12 0x00007ff6877f0cbf in __pyx_pf_6gevent_17_gevent_cgreenlet_8Greenlet_42run (__pyx_v_self=0x7ff686929310) at src/gevent/greenlet.c:14741
#13 __pyx_pw_6gevent_17_gevent_cgreenlet_8Greenlet_43run (__pyx_v_self=<gevent._gevent_cgreenlet.Greenlet at remote 0x7ff686929310>, unused=<optimized out>) at src/gevent/greenlet.c:14521
#14 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<_cython_3_0_0a11.cython_function_or_method at remote 0x7ff687962e40>, args=0x7ffe49b28330, nargsf=1, kwnames=0x0)
    at ./Include/internal/pycore_call.h:92
#15 0x00000000004e09e6 in method_vectorcall (method=<method at remote 0x7ff6861e5a90>, args=0x9dda90 <_PyRuntime+58928>, nargsf=0, kwnames=0x0) at Objects/classobject.c:67
#16 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7ff6861e5a90>, tuple=(), kwargs=0x0) at Objects/call.c:245
#17 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7ff6861e5a90>, args=(), kwargs=0x0) at Objects/call.c:328
#18 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7ff6861e5a90>, args=(), kwargs=0x0) at Objects/call.c:355
#19 0x00007ff695605c70 in g_initialstub (mark=<optimized out>) at src/greenlet/greenlet.c:960
#20 0x00007ff695605f1f in g_switch (kwargs=<optimized out>, args=<optimized out>, target=0x7ff686929310) at src/greenlet/greenlet.c:716
#21 g_switch (target=<optimized out>, args=(), kwargs=0x0) at src/greenlet/greenlet.c:678
#22 0x00007ff686e45010 in ?? ()
#23 0x00007ff687a52510 in ?? ()
#24 0x00007ff687a52510 in ?? ()
#25 0x00000000009cf5c0 in _PyRuntime ()
#26 0x00000000009dda98 in _PyRuntime ()
#27 0x0000000000000000 in ?? ()

Your environment

CPython versions tested on: 3.11.0rc2
Operating system and architecture: Fedora Linux x86_64

Oct 08 '22 23:10 arcivanov

core.python3\x2e11.1000.9a4a42f4641349f28fff1fe56dc227f4.701088.1665271284000000.gz

Oct 09 '22 00:10 arcivanov

Can you describe how to reproduce this issue?

Oct 09 '22 00:10 sweeneyde

Install CPython 3.11.0rc2 with "-O0".
Create and activate VirtualEnv for the test case.
In the venv install PyBuilder (pip install pybuilder)
Clone and enter https://github.com/karellen/geventmp
Run pyb -vX
Observe the failure during integration testing and coredumps (e.g. via coredumpctl).

[INFO]  Reusing target 'test' VEnv in '/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2'
[DEBUG] Executing command: '/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/bin/python3.11' '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py'
[ERROR] Integration test failed: /home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py, exit code 1
File /home/arcivanov/Documents/src/karellen/geventmp/target/reports/integrationtests/monkey_manager_tests:
/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/envs/geventmp-3.11g/lib/python3.11/site-packages/pybuilder/terminal.py:135: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/arcivanov/Documents/src/karellen/geventmp/target/reports/integrationtests/monkey_manager_tests' mode='r' encoding='UTF-8'>
  for line in open(file_name):
ResourceWarning: Enable tracemalloc to get the object allocation traceback

File /home/arcivanov/Documents/src/karellen/geventmp/target/reports/integrationtests/monkey_manager_tests.err:
    python3.11: Objects/object.c:2276: _PyTrash_thread_destroy_chain: Assertion `tstate->trash_delete_nesting == 1' failed.
    F.
    ======================================================================
    FAIL: test_manager_fork (__main__.TestSyncManager.test_manager_fork)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 71, in test_manager_fork
        self.run_manager_test("fork")
      File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 81, in run_manager_test
        self._test_manager(ctx, remote_trace)
      File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 99, in _test_manager
        self.assertEqual(p.exitcode, 0)
    AssertionError: -6 != 0
    
    ----------------------------------------------------------------------
    Ran 2 tests in 322.155s
    
    FAILED (failures=1)
/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/envs/geventmp-3.11g/lib/python3.11/site-packages/pybuilder/terminal.py:135: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/arcivanov/Documents/src/karellen/geventmp/target/reports/integrationtests/monkey_manager_tests.err' mode='r' encoding='UTF-8'>
  for line in open(file_name):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
[INFO]  Running integration test monkey_mp_tests

After having run the #5 at least once, you'll be able to run your tests like: PURE_PYTHON=1 '/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/bin/python3.11' '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py' (your paths will vary).

Oct 09 '22 00:10 arcivanov

op=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), 83), None), 55), None), 13), None), 42), None), 70), None), 7), None), 8), None), 51), None), None), ()), (...)), (...)), None), None), 15), None), 64), None), 94), None), 76), None), None), None), None), None), None), 91), None), 41), None), None), None), None), 31), None), 4), None), None), None), 78), None), 79), None), None), None), None), None), 92), None), 39), None), 23), None), 93), None), 68), None), 47), None), 74), None), 84), None), 72), None), 98), None), 61), None), 90), None), None), 99), 33)

Do you where this linked-list-of-tuples comes from? In particular, which pieces do the two (...) refer to? If you can figure that out, does constructing it from the ground up cause the assertion failure without using any external libraries?

I was trying something like this, but I did not find any assertion failures:

diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
index 173d7c2cb8..97e650ca02 100644
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -5643,7 +5643,121 @@ static PyObject *test_buildvalue_issue38913(PyObject *, PyObject *);
 static PyObject *getargs_s_hash_int(PyObject *, PyObject *, PyObject*);
 static PyObject *getargs_s_hash_int2(PyObject *, PyObject *, PyObject*);

+static PyObject *
+tup(PyObject *a, PyObject *b)
+{
+    PyObject *result = PyTuple_New(2);
+    assert(result);
+    PyTuple_SET_ITEM(result, 0, a);
+    PyTuple_SET_ITEM(result, 1, b);
+    return result;
+}
+
+static PyObject *
+test_destroy_chain(PyObject *self, PyObject *Py_UNUSED(ignored))
+{
+    PyObject *res =
+        tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(
+        tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(
+        tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(
+        tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(tup(
+        tup(tup(tup(tup(
+        NULL,
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(83)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(55)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(13)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(42)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(70)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(7)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(8)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(51)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        PyTuple_New(0)),
+        NULL),
+        NULL),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(15)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(64)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(94)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(76)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(91)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(41)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(31)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(4)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(78)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(79)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(92)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(39)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(23)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(93)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(68)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(47)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(74)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(84)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(72)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(98)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(61)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(90)),
+        Py_NewRef(Py_None)),
+        Py_NewRef(Py_None)),
+        PyLong_FromLong(99)),
+        PyLong_FromLong(33));
+    PyObject *pair;
+    for (pair = res; pair != NULL; pair = PyTuple_GET_ITEM(pair, 0)) {
+        if (PyTuple_GET_ITEM(pair, 1) == NULL) {
+            PyTuple_SET_ITEM(pair, 1, Py_NewRef(PyTuple_GET_ITEM(pair, 0)));
+        }
+    }
+    return res;
+}
+
 static PyMethodDef TestMethods[] = {
+    {"test_destroy_chain", test_destroy_chain, METH_NOARGS},
     {"raise_exception",         raise_exception,                 METH_VARARGS},
     {"raise_memoryerror",       raise_memoryerror,               METH_NOARGS},
     {"set_errno",               set_errno,                       METH_VARARGS},

Maybe @pablogsal has an idea about this.

Oct 11 '22 20:10 sweeneyde

@jamadden can you help? This seems to be somewhere in the Cython code.

Oct 11 '22 20:10 arcivanov

The last call including gevent code is this one (name mangling prefix removed and otherwise shortened):

#11 …...Greenlet__Greenlet__free (__pyx_v_self=__pyx_v_self@entry=0x...) at greenlet.c:14915

i.e., Greenlet.__free().

The context around line 14915 in greenlet.c from gevent-22.8.0 is this; I've marked exactly line 14915.

  /* "src/gevent/greenlet.py":925
 *         except AttributeError:
 *             pass
 *         self.args = ()             # <<<<<<<<<<<<<<
 *         self.kwargs.clear()
 * 
 */
  __Pyx_INCREF(__pyx_empty_tuple);
  __Pyx_GIVEREF(__pyx_empty_tuple);
  __Pyx_GOTREF(__pyx_v_self->args);
  __Pyx_DECREF(__pyx_v_self->args);  // <— 14915
  __pyx_v_self->args = __pyx_empty_tuple;

As you can see, this is the code Cython produced for self.args = () inside the function Greenlet.__free. The function Greenlet.__free is called after a greenlet finishes running (there is another place that can call it, but that's by far the most common).

The args property of a Greenlet is a tuple given to the constructor of the greenlet, passed to the function the greenlet is supposed to run, and then it is freed in this manner.

Cython exposes args as a readonly property, so it shouldn't be possible to assign to it once the greenlet has been constructed. gevent itself doesn't manipulate args, other than to pass it along to the function.

Are you able to run your code without most of the gevent C extensions, in pure-python mode, using the environment variable PURE_PYTHON=1, and if so, does that change anything? That's the sledge hammer approach, of course, and would require further narrowing if successful.

And/or, does anything change if you run your program with the environment variable GEVENT_TRACK_GREENLET_TREE set to false, and does that change anything? I ask because getting the frame handling right for that feature was the trickiest part of supporting Python 3.11 and early versions had refcounting bugs, which can and did lead to apparently unrelated issues.

Oct 11 '22 21:10 jamadden

@jamadden Thank you so much. Running with PURE_PYTHON=1 resulted in a different backtrace but the same issue:

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f716d08ecb3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007f716d03e9c6 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f716d0287f4 in __GI_abort () at abort.c:79
#4  0x00007f716d02871b in __assert_fail_base (fmt=0x7f716d1bbda0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2276, 
    function=<optimized out>) at assert.c:92
#5  0x00007f716d037576 in __GI___assert_fail (assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2276, function=0x786cc0 <__PRETTY_FUNCTION__.3> "_PyTrash_thread_destroy_chain")
    at assert.c:101
#6  0x0000000000549867 in _PyTrash_thread_destroy_chain () at Objects/object.c:2276
#7  0x000000000054992b in _PyTrash_end (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2301
#8  0x000000000055c1eb in tupledealloc (op=0x7f715e1cfac0) at Objects/tupleobject.c:215
#9  0x0000000000549ba2 in _Py_Dealloc (
    op=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), 17), None), 59), None), None), None), None), 31), None), 23), None), None), ()), None), None), None), None), None), None), 81), None), 64), None), None), None), 66), None), None), None), (...)), None), (...)), None), None), None), None), 56), None), 18), None), None), None), None), None), None), None), 89), None), 61), None), None), None), None), None), None), None), None), 15), None), 50), 26)) at Objects/object.c:2389
#10 0x00000000005272d8 in Py_DECREF (filename=0x77e260 "./Include/object.h", lineno=602, 
    op=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), 17), None), 59), None), None), None), None), 31), None), 23), None), None), ()), None), None), None), None), None), None), 81), None), 64), None), None), None), 66), None), None), None), (...)), None), (...)), None), None), None), None), 56), None), 18), None), None), None), None), None), None), None), 89), None), 61), None), None), None), None), None), None), None), None), 15), None), 50), 26)) at ./Include/object.h:527
#11 0x0000000000527326 in Py_XDECREF (
    op=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), 17), None), 59), None), None), None), None), 31), None), 23), None), None), ()), None), None), None), None), None), None), 81), None), 64), None), None), None), 66), None), None), None), (...)), None), (...)), None), None), None), None), 56), None), 18), None), None), None), None), None), None), None), 89), None), 61), None), None), None), None), None), None), None), None), 15), None), 50), 26)) at ./Include/object.h:602
#12 0x000000000052b369 in insertdict (mp=0x7f715e1ca7b0, key='args', hash=9178157314151281872, value=()) at Objects/dictobject.c:1304
#13 0x000000000052cee8 in _PyDict_SetItem_Take2 (mp=0x7f715e1ca7b0, key='args', value=()) at Objects/dictobject.c:1886
#14 0x000000000052cfa0 in PyDict_SetItem (
    op={'args': (), 'kwargs': {}, 'value': None, '_start_event': <_dummy_event at remote 0x7f715f5137a0>, '_notifier': <gevent.libev.corecext.callback at remote 0x7f715e1cf9d0>, '_formatted_info': None, '_links': [], '_ident': None, '_exc_info': (None, None, None), 'spawning_greenlet': <weakref.ReferenceType at remote 0x7f715e04c4b0>, 'spawn_tree_locals': {}, 'spawning_stack': <_Frame at remote 0x7f715ea8b480>, '_gevent_local_localimpl_140124886819216': <weakref.ReferenceType at remote 0x7f715e0598d0>}, key='args', value=()) at Objects/dictobject.c:1906
#15 0x00000000005354c2 in _PyObjectDict_SetItem (tp=0xfaa390, dictptr=0x7f715e1e6848, key='args', value=()) at Objects/dictobject.c:5668
#16 0x000000000054890d in _PyObject_GenericSetAttrWithDict (
    obj=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7f715f5137a0>, _notifier=<gevent.libev.corecext.callback at remote 0x7f715e1cf9d0>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=<weakref.ReferenceType at remote 0x7f715e04c4b0>, spawn_tree_locals={}, spawning_stack=<_Frame at remote 0x7f715ea8b480>, _gevent_local_localimpl_140124886819216=<weakref.ReferenceType at remote 0x7f715e0598d0>) at remote 0x7f715e1e67b0>, name='args', value=(), dict=0x0) at Objects/object.c:1422
#17 0x0000000000548a54 in PyObject_GenericSetAttr (
    obj=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7f715f5137a0>, _notifier=<gevent.libev.corecext.callback at remote 0x7f715e1cf9d0>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=<weakref.ReferenceType at remote 0x7f715e04c4b0>, spawn_tree_locals={}, spawning_stack=<_Frame at remote 0x7f715ea8b480>, _gevent_local_localimpl_140124886819216=<weakref.ReferenceType at remote 0x7f715e0598d0>) at remote 0x7f715e1e67b0>, name='args', value=()) at Objects/object.c:1456
#18 0x0000000000547ab4 in PyObject_SetAttr (
    v=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7f715f5137a0>, _notifier=<gevent.libev.corecext.callback at remote 0x7f715e1cf9d0>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=<weakref.ReferenceType at remote 0x7f715e04c4b0>, spawn_tree_locals={}, spawning_stack=<_Frame at remote 0x7f715ea8b480>, _gevent_local_localimpl_140124886819216=<weakref.ReferenceType at remote 0x7f715e0598d0>) at remote 0x7f715e1e67b0>, name='args', value=()) at Objects/object.c:1028
#19 0x0000000000610593 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7f715de780a0, throwflag=0) at Python/ceval.c:2889
#20 0x0000000000605fec in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7f715de78020, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#21 0x0000000000623420 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7f715f5203c0, locals=0x0, args=0x7fff8b6527c0, argcount=1, kwnames=0x0) at Python/ceval.c:6428
#22 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7f715f5203c0>, stack=0x7fff8b6527c0, nargsf=1, kwnames=0x0) at Objects/call.c:393
#23 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7f715f5203c0>, args=0x7fff8b6527c0, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#24 0x00000000004e09e6 in method_vectorcall (method=<method at remote 0x7f715e1f6570>, args=0x9dda90 <_PyRuntime+58928>, nargsf=0, kwnames=0x0) at Objects/classobject.c:67
#25 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7f715e1f6570>, tuple=(), kwargs=0x0) at Objects/call.c:245
#26 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7f715e1f6570>, args=(), kwargs=0x0) at Objects/call.c:328
#27 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7f715e1f6570>, args=(), kwargs=0x0) at Objects/call.c:355
#28 0x00007f715f542c70 in g_initialstub (mark=<optimized out>) at src/greenlet/greenlet.c:960
#29 0x00007f715f542f1f in g_switch (kwargs=<optimized out>, args=<optimized out>, target=0x7f715e1e67b0) at src/greenlet/greenlet.c:716
#30 g_switch (target=<optimized out>, args=(), kwargs=0x0) at src/greenlet/greenlet.c:678
#31 0x00007f715db10ad0 in ?? ()
#32 0x00007f715db11250 in ?? ()
#33 0x00007f715db11250 in ?? ()
#34 0x00000000009cf5c0 in _PyRuntime ()
#35 0x00000000009dda98 in _PyRuntime ()
#36 0x0000000000000000 in ?? ()

core.python3\x2e11.1000.a73c765f59ff4f388aea5262a29a16b9.43148.1665528851000000.gz

Oct 11 '22 22:10 arcivanov

This is non-pure GEVENT_TRACK_GREENLET_TREE=0. Notice the tuple chain op in #9 seems shallower.

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44            return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f58cec8ecb3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007f58cec3e9c6 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f58cec287f4 in __GI_abort () at abort.c:79
#4  0x00007f58cec2871b in __assert_fail_base (fmt=0x7f58cedbbda0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2276, 
    function=<optimized out>) at assert.c:92
#5  0x00007f58cec37576 in __GI___assert_fail (assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2276, function=0x786cc0 <__PRETTY_FUNCTION__.3> "_PyTrash_thread_destroy_chain")
    at assert.c:101
#6  0x0000000000549867 in _PyTrash_thread_destroy_chain () at Objects/object.c:2276
#7  0x000000000054992b in _PyTrash_end (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2301
#8  0x000000000055c1eb in tupledealloc (op=0x7f58bfdd7f70) at Objects/tupleobject.c:215
#9  0x0000000000549ba2 in _Py_Dealloc (
    op=(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), 68), None), None), None), 65), None), 25), None), 59), None), 69), None), 35), None), 45), None), 74), None), 23), None), 53), None), None), None), None), None), 57), None), 28), None), 75), None), None), None), None), 17), None), 90), None), 71), None), None), None), 43), None), 26), None), 51), None), 54), None), None), None), None), None), None), None), 66), None), 89), None), 87), None), 91), None), 39), None), 95), None), 72), None), 92), None), None), None), 31), None), 70), None), 84), None), None), None), None), None), None), 97), 65)) at Objects/object.c:2389
#10 0x00007f58c10efd1d in Py_DECREF (op=<optimized out>) at /home/arcivanov/.pyenv/versions/3.11.0rc2/include/python3.11/object.h:538
#11 __pyx_f_6gevent_17_gevent_cgreenlet_8Greenlet__Greenlet__free (__pyx_v_self=__pyx_v_self@entry=0x7f58bfdde490) at src/gevent/greenlet.c:14915
#12 0x00007f58c1104cbf in __pyx_pf_6gevent_17_gevent_cgreenlet_8Greenlet_42run (__pyx_v_self=0x7f58bfdde490) at src/gevent/greenlet.c:14741
#13 __pyx_pw_6gevent_17_gevent_cgreenlet_8Greenlet_43run (__pyx_v_self=<gevent._gevent_cgreenlet.Greenlet at remote 0x7f58bfdde490>, unused=<optimized out>) at src/gevent/greenlet.c:14521
#14 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<_cython_3_0_0a11.cython_function_or_method at remote 0x7f58c127ae40>, args=0x7fff26a54aa0, nargsf=1, kwnames=0x0)
    at ./Include/internal/pycore_call.h:92
#15 0x00000000004e09e6 in method_vectorcall (method=<method at remote 0x7f58bfdf28d0>, args=0x9dda90 <_PyRuntime+58928>, nargsf=0, kwnames=0x0) at Objects/classobject.c:67
#16 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7f58bfdf28d0>, tuple=(), kwargs=0x0) at Objects/call.c:245
#17 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7f58bfdf28d0>, args=(), kwargs=0x0) at Objects/call.c:328
#18 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7f58bfdf28d0>, args=(), kwargs=0x0) at Objects/call.c:355
#19 0x00007f58cee0bc70 in g_initialstub (mark=<optimized out>) at src/greenlet/greenlet.c:960
#20 0x00007f58cee0bf1f in g_switch (kwargs=<optimized out>, args=<optimized out>, target=0x7f58bfdde490) at src/greenlet/greenlet.c:716
#21 g_switch (target=<optimized out>, args=(), kwargs=0x0) at src/greenlet/greenlet.c:678
#22 0x00007f58c0673a70 in ?? ()
#23 0x00007f58c0926930 in ?? ()
#24 0x00007f58c0926930 in ?? ()
#25 0x00000000009cf5c0 in _PyRuntime ()
#26 0x00000000009dda98 in _PyRuntime ()
#27 0x0000000000000000 in ?? ()

core.python3\x2e11.1000.a73c765f59ff4f388aea5262a29a16b9.44630.1665529256000000.gz

Oct 11 '22 23:10 arcivanov

PURE_PYTHON=1 GEVENT_TRACK_GREENLET_TREE=0 '/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/bin/python3.11' '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py'

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f704dc8ecb3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007f704dc3e9c6 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f704dc287f4 in __GI_abort () at abort.c:79
#4  0x00007f704dc2871b in __assert_fail_base (fmt=0x7f704ddbbda0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2276, 
    function=<optimized out>) at assert.c:92
#5  0x00007f704dc37576 in __GI___assert_fail (assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2276, function=0x786cc0 <__PRETTY_FUNCTION__.3> "_PyTrash_thread_destroy_chain")
    at assert.c:101
#6  0x0000000000549867 in _PyTrash_thread_destroy_chain () at Objects/object.c:2276
#7  0x000000000054992b in _PyTrash_end (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2301
#8  0x000000000055c1eb in tupledealloc (op=0x7f703f392210) at Objects/tupleobject.c:215
#9  0x0000000000549ba2 in _Py_Dealloc (
    op=(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), ()), (...)), None), None), 23), None), None), None), None), None), (...)), None), None), None), None), 55), None), None), (...)), None), None), None), None), 57), None), 29), None), None), None), 42), None), 38), None), 30), None), 67), None), 34), None), None), None), None), None), None), None), 13), (...)), (...)), (...)), None), None), None), (...)), (...)), (...)), None), None), 71), None), None), 31), None), 72), None), 90), None), 68), 76)) at Objects/object.c:2389
#10 0x00000000005272d8 in Py_DECREF (filename=0x77e260 "./Include/object.h", lineno=602, 
    op=(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), ()), (...)), None), None), 23), None), None), None), None), None), (...)), None), None), None), None), 55), None), None), (...)), None), None), None), None), 57), None), 29), None), None), None), 42), None), 38), None), 30), None), 67), None), 34), None), None), None), None), None), None), None), 13), (...)), (...)), (...)), None), None), None), (...)), (...)), (...)), None), None), 71), None), None), 31), None), 72), None), 90), None), 68), 76)) at ./Include/object.h:527
#11 0x0000000000527326 in Py_XDECREF (
    op=(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), ()), (...)), None), None), 23), None), None), None), None), None), (...)), None), None), None), None), 55), None), None), (...)), None), None), None), None), 57), None), 29), None), None), None), 42), None), 38), None), 30), None), 67), None), 34), None), None), None), None), None), None), None), 13), (...)), (...)), (...)), None), None), None), (...)), (...)), (...)), None), None), 71), None), None), 31), None), 72), None), 90), None), 68), 76)) at ./Include/object.h:602
#12 0x000000000052b369 in insertdict (mp=0x7f703f38d970, key='args', hash=7479772573573864687, value=()) at Objects/dictobject.c:1304
#13 0x000000000052cee8 in _PyDict_SetItem_Take2 (mp=0x7f703f38d970, key='args', value=()) at Objects/dictobject.c:1886
#14 0x000000000052cfa0 in PyDict_SetItem (
    op={'args': (), 'kwargs': {}, 'value': None, '_start_event': <_dummy_event at remote 0x7f70403b1b80>, '_notifier': <gevent.libev.corecext.callback at remote 0x7f703ecd0b90>, '_formatted_info': None, '_links': [], '_ident': None, '_exc_info': (None, None, None), 'spawning_greenlet': None, 'spawn_tree_locals': None, 'spawning_stack': None, '_gevent_local_localimpl_140120073666048': <weakref.ReferenceType at remote 0x7f703ecfde10>}, key='args', value=())
    at Objects/dictobject.c:1906
#15 0x00000000005354c2 in _PyObjectDict_SetItem (tp=0x1afc5b0, dictptr=0x7f703f5db8e8, key='args', value=()) at Objects/dictobject.c:5668
#16 0x000000000054890d in _PyObject_GenericSetAttrWithDict (
    obj=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7f70403b1b80>, _notifier=<gevent.libev.corecext.callback at remote 0x7f703ecd0b90>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140120073666048=<weakref.ReferenceType at remote 0x7f703ecfde10>) at remote 0x7f703f5db850>, name='args', value=(), dict=0x0)
    at Objects/object.c:1422
#17 0x0000000000548a54 in PyObject_GenericSetAttr (
    obj=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7f70403b1b80>, _notifier=<gevent.libev.corecext.callback at remote 0x7f703ecd0b90>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140120073666048=<weakref.ReferenceType at remote 0x7f703ecfde10>) at remote 0x7f703f5db850>, name='args', value=())
    at Objects/object.c:1456
#18 0x0000000000547ab4 in PyObject_SetAttr (
    v=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7f70403b1b80>, _notifier=<gevent.libev.corecext.callback at remote 0x7f703ecd0b90>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140120073666048=<weakref.ReferenceType at remote 0x7f703ecfde10>) at remote 0x7f703f5db850>, name='args', value=())
    at Objects/object.c:1028
#19 0x0000000000610593 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7f703eb800a0, throwflag=0) at Python/ceval.c:2889
#20 0x0000000000605fec in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7f703eb80020, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#21 0x0000000000623420 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7f704023c3c0, locals=0x0, args=0x7ffcdf5bddf0, argcount=1, kwnames=0x0) at Python/ceval.c:6428
#22 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7f704023c3c0>, stack=0x7ffcdf5bddf0, nargsf=1, kwnames=0x0) at Objects/call.c:393
#23 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7f704023c3c0>, args=0x7ffcdf5bddf0, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#24 0x00000000004e09e6 in method_vectorcall (method=<method at remote 0x7f703f805e50>, args=0x9dda90 <_PyRuntime+58928>, nargsf=0, kwnames=0x0) at Objects/classobject.c:67
#25 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7f703f805e50>, tuple=(), kwargs=0x0) at Objects/call.c:245
#26 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7f703f805e50>, args=(), kwargs=0x0) at Objects/call.c:328
#27 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7f703f805e50>, args=(), kwargs=0x0) at Objects/call.c:355
#28 0x00007f704dbbbc70 in g_initialstub (mark=<optimized out>) at src/greenlet/greenlet.c:960
#29 0x00007f704dbbbf1f in g_switch (kwargs=<optimized out>, args=<optimized out>, target=0x7f703f5db850) at src/greenlet/greenlet.c:716
#30 g_switch (target=<optimized out>, args=(), kwargs=0x0) at src/greenlet/greenlet.c:678
#31 0x00007f703ed53d10 in ?? ()
#32 0x00007f703ed52030 in ?? ()
#33 0x00007f703ed52030 in ?? ()
#34 0x00000000009cf5c0 in _PyRuntime ()
#35 0x00000000009dda98 in _PyRuntime ()
#36 0x0000000000000000 in ?? ()

core.python3\x2e11.1000.a73c765f59ff4f388aea5262a29a16b9.46122.1665529657000000.gz

Oct 11 '22 23:10 arcivanov

With the output so far, there is no way to know this is something in CPython or in some of the C extensions (or Cython) you are using. Could you please provide a reproducer without any 3rd party libraries?

Oct 12 '22 00:10 pablogsal

With the output so far, there is no way to know this is something in CPython or in some of the C extensions (or Cython) you are using. Could you please provide a reproducer without any 3rd party libraries?

Even with a bt like this? https://github.com/python/cpython/issues/98110#issuecomment-1275378736

If I could provider a reproducer without 3rd-party libraries I would. This was found automatically as part of CI regression testing.

Oct 12 '22 01:10 arcivanov

Even with a bt like this? #98110 (comment)

Yes, that backtrace unfortunately doesn't tell us much. The problem may have happened because some C extension is misusing the C-API and is corrupting memory or the internal structures. In fact, that backtrace just tells us that something crashes in the trashcan mechanism when destroying an object, which is the only way to invoke the trashcan mechanism. It tell us nothing on how the objects involve were created or if something is corrupting anything.

Oct 12 '22 01:10 pablogsal

If I could provider a reproducer without 3rd-party libraries I would. This was found automatically as part of CI regression testing.

I understand, but please understand that for us is very difficult to locate where the problem is (if there is a problem at all in CPython) if we need to debug across several libraries we are not familiar with. For us is not even clear that the problem is in CPython so is unlikely that we will spend a lot of time trying to find out without an easier reproducer we can work with.

Oct 12 '22 01:10 pablogsal

I understand, but please understand that for us is very difficult to locate where the problem is (if there is a problem at all in CPython) if we need to debug across several libraries we are not familiar with.

I fully understand, that's why the library maintainer is also on this thread to see if the issue could be in the source library. So perhaps in could help brainstorming this a little bit: what is the nature of this assertion and what memory corruption (if it is memory corruption?) could cause such an assertion?

https://github.com/python/cpython/blob/9b12084aca1a2072f0333dbc562e52d8dc0235bd/Objects/object.c#L2248-L2260

Oct 12 '22 02:10 arcivanov

That is checking that the trashcan mechanism is not called recursively inside itself. This is, no object destruction triggers the trashcan while the trashcan is running.

Oct 12 '22 02:10 pablogsal

Notice that that check is per thread state. My guess is that greenlet is doing something weird switching states in the middle of a trashcan run into sone arbitrary code that also calls the trashcan, which would be very not supported. Indeed, you even can see g_switch in your back trace.

Oct 12 '22 02:10 pablogsal

Notice that that check is per thread state. My guess is that greenlet is doing something weird switching states in the middle of a trashcan run into sone arbitrary code that also calls the trashcan, which would be very not supported. Indeed, you even can see g_switch in your back trace.

Good point. @jamadden?

That said here is the <=0: https://github.com/python/cpython/blob/9b12084aca1a2072f0333dbc562e52d8dc0235bd/Objects/object.c#L2300

Is it actually possible (and valid) for trash_delete_nesting to go negative? Neither greenlet nor gevent actually mutate trash_delete_nesting directly at all and switching greenlets without arriving at a switch point is not possible - it has to be cooperative (i.e. block on I/O or yield).

Oct 12 '22 02:10 arcivanov

@pablogsal

Additional question:

destroy_chain is static. Why is tstate simply not passed? Is there any reason for it?

void
_PyTrash_end(PyThreadState *tstate)
{
    --tstate->trash_delete_nesting;
    if (tstate->trash_delete_later && tstate->trash_delete_nesting <= 0) {
        _PyTrash_thread_destroy_chain();
    }
}

static void
_PyTrash_thread_destroy_chain(void)
{
    PyThreadState *tstate = _PyThreadState_GET();
    ...

Same with _start and deposit_ combo. Any reason for it?

Oct 12 '22 02:10 arcivanov

So, quick experiment.

Applied the following patch and rebuilt:

--- Objects/object.c    2022-09-11 15:23:30.000000000 -0400
+++ Objects/object.c.new        2022-10-11 23:03:15.199091458 -0400
@@ -2229,9 +2229,8 @@
  * object, with refcount 0.  Py_DECREF must already have been called on it.
  */
 static void
-_PyTrash_thread_deposit_object(PyObject *op)
+_PyTrash_thread_deposit_object(PyThreadState *tstate, PyObject *op)
 {
-    PyThreadState *tstate = _PyThreadState_GET();
     _PyObject_ASSERT(op, _PyObject_IS_GC(op));
     _PyObject_ASSERT(op, !_PyObject_GC_IS_TRACKED(op));
     _PyObject_ASSERT(op, Py_REFCNT(op) == 0);
@@ -2242,9 +2241,8 @@
 /* Deallocate all the objects in the gcstate->trash_delete_later list.
  * Called when the call-stack unwinds again. */
 static void
-_PyTrash_thread_destroy_chain(void)
+_PyTrash_thread_destroy_chain(PyThreadState *tstate)
 {
-    PyThreadState *tstate = _PyThreadState_GET();
     /* We need to increase trash_delete_nesting here, otherwise,
        _PyTrash_thread_destroy_chain will be called recursively
        and then possibly crash.  An example that may crash without
@@ -2285,7 +2283,7 @@
     if (tstate->trash_delete_nesting >= _PyTrash_UNWIND_LEVEL) {
         /* Store the object (to be deallocated later) and jump past
          * Py_TRASHCAN_END, skipping the body of the deallocator */
-        _PyTrash_thread_deposit_object(op);
+        _PyTrash_thread_deposit_object(tstate, op);
         return 1;
     }
     ++tstate->trash_delete_nesting;
@@ -2298,7 +2296,7 @@
 {
     --tstate->trash_delete_nesting;
     if (tstate->trash_delete_later && tstate->trash_delete_nesting <= 0) {
-        _PyTrash_thread_destroy_chain();
+        _PyTrash_thread_destroy_chain(tstate);
     }
 }

There is still an assertion failure:

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f78ddc8ecb3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007f78ddc3e9c6 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f78ddc287f4 in __GI_abort () at abort.c:79
#4  0x00007f78ddc2871b in __assert_fail_base (fmt=0x7f78dddbbda0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2274, 
    function=<optimized out>) at assert.c:92
#5  0x00007f78ddc37576 in __GI___assert_fail (assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2274, function=0x786cc0 <__PRETTY_FUNCTION__.3> "_PyTrash_thread_destroy_chain")
    at assert.c:101
#6  0x000000000054985d in _PyTrash_thread_destroy_chain (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2274
#7  0x000000000054992f in _PyTrash_end (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2299
#8  0x000000000055c1ef in tupledealloc (op=0x7f78cebc5630) at Objects/tupleobject.c:215
#9  0x0000000000549ba6 in _Py_Dealloc (
    op=(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), None), None), None), None), None), ()), None), None), (...)), None), (...)), None), None), 16), (...)), None), None), None), None), None), None), None), None), None), None), 43), None), None), (...)), None), None), 58), None), 34), None), None), None), 45), None), 8), None), None), None), 47), None), None), None), None), None), None), 57), None), None), None), 20), None), None), None), None), None), None), None), 38), None), None), None), 64), 30)) at Objects/object.c:2387
#10 0x00000000005272d8 in Py_DECREF (filename=0x77e260 "./Include/object.h", lineno=602, 
    op=(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), None), None), None), None), None), ()), None), None), (...)), None), (...)), None), None), 16), (...)), None), None), None), None), None), None), None), None), None), None), 43), None), None), (...)), None), None), 58), None), 34), None), None), None), 45), None), 8), None), None), None), 47), None), None), None), None), None), None), 57), None), None), None), 20), None), None), None), None), None), None), None), 38), None), None), None), 64), 30)) at ./Include/object.h:527
#11 0x0000000000527326 in Py_XDECREF (
    op=(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), None), None), None), None), None), ()), None), None), (...)), None), (...)), None), None), 16), (...)), None), None), None), None), None), None), None), None), None), None), 43), None), None), (...)), None), None), 58), None), 34), None), None), None), 45), None), 8), None), None), None), 47), None), None), None), None), None), None), 57), None), None), None), 20), None), None), None), None), None), None), None), 38), None), None), None), 64), 30)) at ./Include/object.h:602
#12 0x000000000052b369 in insertdict (mp=0x7f78cedeca70, key='args', hash=-5686414954309761599, value=()) at Objects/dictobject.c:1304
#13 0x000000000052cee8 in _PyDict_SetItem_Take2 (mp=0x7f78cedeca70, key='args', value=()) at Objects/dictobject.c:1886
#14 0x000000000052cfa0 in PyDict_SetItem (
    op={'args': (), 'kwargs': {}, 'value': None, '_start_event': <_dummy_event at remote 0x7f78d0113700>, '_notifier': <gevent.libev.corecext.callback at remote 0x7f78cebdda40>, '_formatted_info': None, '_links': [], '_ident': None, '_exc_info': (None, None, None), 'spawning_greenlet': None, 'spawn_tree_locals': None, 'spawning_stack': None, '_gevent_local_localimpl_140156843318800': <weakref.ReferenceType at remote 0x7f78cec2d470>}, key='args', value=())
    at Objects/dictobject.c:1906
#15 0x00000000005354c2 in _PyObjectDict_SetItem (tp=0x1d495b0, dictptr=0x7f78cedd7028, key='args', value=()) at Objects/dictobject.c:5668
#16 0x000000000054890d in _PyObject_GenericSetAttrWithDict (
    obj=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7f78d0113700>, _notifier=<gevent.libev.corecext.callback at remote 0x7f78cebdda40>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140156843318800=<weakref.ReferenceType at remote 0x7f78cec2d470>) at remote 0x7f78cedd6f90>, name='args', value=(), dict=0x0)
    at Objects/object.c:1422
#17 0x0000000000548a54 in PyObject_GenericSetAttr (
    obj=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7f78d0113700>, _notifier=<gevent.libev.corecext.callback at remote 0x7f78cebdda40>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140156843318800=<weakref.ReferenceType at remote 0x7f78cec2d470>) at remote 0x7f78cedd6f90>, name='args', value=())
    at Objects/object.c:1456
#18 0x0000000000547ab4 in PyObject_SetAttr (
    v=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7f78d0113700>, _notifier=<gevent.libev.corecext.callback at remote 0x7f78cebdda40>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140156843318800=<weakref.ReferenceType at remote 0x7f78cec2d470>) at remote 0x7f78cedd6f90>, name='args', value=())
    at Objects/object.c:1028
#19 0x0000000000610597 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7f78cea440a0, throwflag=0) at Python/ceval.c:2889
#20 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7f78cea44020, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#21 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7f78d01203c0, locals=0x0, args=0x7ffc3d025f20, argcount=1, kwnames=0x0) at Python/ceval.c:6428
#22 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7f78d01203c0>, stack=0x7ffc3d025f20, nargsf=1, kwnames=0x0) at Objects/call.c:393
#23 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7f78d01203c0>, args=0x7ffc3d025f20, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#24 0x00000000004e09e6 in method_vectorcall (method=<method at remote 0x7f78cededbb0>, args=0x9dda90 <_PyRuntime+58928>, nargsf=0, kwnames=0x0) at Objects/classobject.c:67
#25 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7f78cededbb0>, tuple=(), kwargs=0x0) at Objects/call.c:245
#26 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7f78cededbb0>, args=(), kwargs=0x0) at Objects/call.c:328
#27 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7f78cededbb0>, args=(), kwargs=0x0) at Objects/call.c:355
#28 0x00007f78d0142c70 in g_initialstub (mark=<optimized out>) at src/greenlet/greenlet.c:960
#29 0x00007f78d0142f1f in g_switch (kwargs=<optimized out>, args=<optimized out>, target=0x7f78cedd6f90) at src/greenlet/greenlet.c:716
#30 g_switch (target=<optimized out>, args=(), kwargs=0x0) at src/greenlet/greenlet.c:678
#31 0x00007f78ce92c290 in ?? ()
#32 0x00007f78ce92c410 in ?? ()
#33 0x00007f78ce92c410 in ?? ()
#34 0x00000000009cf5c0 in _PyRuntime ()
#35 0x00000000009dda98 in _PyRuntime ()
#36 0x0000000000000000 in ?? ()

Notice, however, this:

#6  0x000000000054985d in _PyTrash_thread_destroy_chain (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2274
#7  0x000000000054992f in _PyTrash_end (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2299

Because tstate is now passed via the patch, it is guaranteed that both _end and _destroy_chain are talking to the same tstate with no possible switching.

Oct 12 '22 03:10 arcivanov

destroy_chain is static. Why is tstate simply not passed? Is there any reason for it?

Not that I can think of, probably the evolution of the API across different modifications.

Because tstate is now passed via the patch, it is guaranteed that both _end and _destroy_chain are talking to the same tstate with no possible switching.

I am not sure of that. The destructors being invoked in the trashcan can still cause switching underneath that code. In any case, this is just extending my initial guess that something in greenlet/gevent is doing something nasty, which remains unproved so I cannot reason around it. As I don't have a simple reproducer I can sadly only speculate, which is not very helpful.

Unfortunately, I won't be able to dedicate more time to this issue until there is a reproducer without 3rd party libraries, especially without greenlet/gevent, or alternatively, until someone finds that indeed the problem is in CPython.

This also means that issue won't be blocking the 3.11 final release scheduled for the 24th of October as is not confirmed that this is even an issue within CPython.

Something you can try to do is bisect the issue against all commits between 3.10 and 3.11.0rc2. Maybe this points out to something that we changed internally so we can reason about it better.

Oct 12 '22 13:10 pablogsal

@pablogsal

My guess is that greenlet is doing something weird switching states in the middle of a trashcan run into sone arbitrary code that also calls the trashcan, which would be very not supported.

@arcivanov

Neither greenlet nor gevent actually mutate trash_delete_nesting directly at all and switching greenlets without arriving at a switch point is not possible - it has to be cooperative

Agreed on both points. Switching is explicit. Could some object be switching during its __del__? I suppose so, but that's highly discouraged. I'm not that familiar with the trashcan, but normally if something crashes in __del__, you see slot_tp_finalize or similar in the C stack, which I don't here.

@pablogsal

Indeed, you even can see g_switch in your back trace.

Yes. That only means that this is not the main greenlet for the thread. That's always at the bottom of the backtrace for any non-main greenlet. And it makes sense in context with the rest of the backtrace. The greenlet that was switched to finished (is exiting its run() method), and is now in a finally block cleaning up. Roughly, the basic outline would look something like this:

from greenlet import greenlet # low-level primitives, must be implemented in native code
class Greenlet(greenlet): # high-level Pythonic helpers, can be implemented in Python
    def __init__(self, func, args):
      super().__init__()
      self.func = func
      self.args = args

   def run(self): # The entry point from greenlet.greenlet; called the first time we are switched to
      try:
          self.func(*self.args)
      finally:
          # release resources 
          self.args = () # <— This is where we're crashing

def a_func(…): … 

g = Greenlet(a_func, make_some_args())
g.switch() # <— crashes here: g.run() -> a_func(*make_some_args()) -> g.args = ()

Since we're crashing deallocating self.args, I think that @sweeneyde is right, and it would help a lot to know what the contents of self.args are; those are created and passed by the application or framework (make_some_args in this example).

It's pretty easy to modify the Greenlet class to print the args that are given to it, and where they came from.

Unfortunately, following the reproduction steps hasn't worked for me so far. By that I mean that the tests run without crashing on my Intel macOS 12.6; they just don't pass.

# PURE_PYTHON=1 /geventmp/target/venv/test/cpython-3.11.0.candidate.2/bin/python src/integrationtest/python/monkey_manager_tests.py  -v
test_manager_fork (__main__.TestSyncManager.test_manager_fork) ... 
… eliding several errors like this one:
Traceback (most recent call last):
  File "//python3.11/multiprocessing/managers.py", line 814, in _callmethod
    conn = self._tls.connection
           ^^^^^^^^^^^^^^^^^^^^
  File "/src/gevent/local.py", line 467, in __getattribute__
    raise AttributeError("%r object has no attribute '%s'"
created with args ()
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
created with args ()
  File "//src/gevent/greenlet.py", line 910, in run
    result = self._run(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "//geventmp/src/integrationtest/python/_mp_manager_server.py", line 107, in process_queue
    q.put(put_string)
  File "<string>", line 2, in put
  File "//python3.11/multiprocessing/managers.py", line 818, in _callmethod
    self._connect()
  File "//python3.11/multiprocessing/managers.py", line 805, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "//python3.11/multiprocessing/connection.py", line 506, in Client
    c = SocketClient(address)
        ^^^^^^^^^^^^^^^^^^^^^
  File "//python3.11/multiprocessing/connection.py", line 634, in SocketClient
    s.connect(address)
  File "/src/gevent/_socketcommon.py", line 628, in connect
    raise _SocketError(result, strerror(result))
ConnectionRefusedError: [Errno 61] Connection refused
...
2022-10-12T11:28:51Z <Greenlet at 0x103e51390: process_queue(<AutoProxy[Queue] object, typeid 'Queue' at 0x103e, 33)> failed with ConnectionRefusedError
FAIL

======================================================================
FAIL: test_manager_fork (__main__.TestSyncManager.test_manager_fork)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 71, in test_manager_fork
    self.run_manager_test("fork")
  File "//geventmp/src/integrationtest/python/monkey_manager_tests.py", line 81, in run_manager_test
    self._test_manager(ctx, remote_trace)
  File "//geventmp/src/integrationtest/python/monkey_manager_tests.py", line 98, in _test_manager
    self.assertFalse(p.is_alive())
AssertionError: True is not false

======================================================================
FAIL: test_manager_spawn (__main__.TestSyncManager.test_manager_spawn)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 74, in test_manager_spawn
    self.run_manager_test("spawn")
  File "//geventmp/src/integrationtest/python/monkey_manager_tests.py", line 81, in run_manager_test
    self._test_manager(ctx, remote_trace)
  File "//geventmp/src/integrationtest/python/monkey_manager_tests.py", line 98, in _test_manager
    self.assertFalse(p.is_alive())
AssertionError: True is not false

----------------------------------------------------------------------
Ran 2 tests in 616.040s
…

Now, so far I don't think I've been using a debug build of CPython, so it could be that the corruption exists, it's just not enough to kill things on its own without assert being enabled. But I had modified my Greenlet class to print arguments, and saw nothing like that monster tuple. I'll try with a debug build, but I'm not super hopeful.

@arcivanov As I'm unable to reproduce so far, can you try modifying your Greenlet class (PURE_PYTHON mode, naturally) to print its arguments when it is created, so you can see when a greenlet is created with such a nested tuple? You could also try taking out the self.args = () from the finally block in Greenlet.run (or indeed, remove the finally block altogether) to see if the crash is shadowing some other exception that might be important.

Oct 12 '22 14:10 jamadden

you see slot_tp_finalize or similar in the C stack, which I don't here.

Yeah but this this a more complex case. What's happening here is that the trashcan allows for linearised deferred destruction. It basically schedules a bunch of tp_dealloc to be called in a row to avoid recursive destruction that can overwhelm the C stack. This means that we are likely seeing the effects of something that may already happened or that happened in another order. This means that also the normal call stack for destruction is different.

The error is basically complaining that something is calling the trashcan while the trashcan is being already being processed. Based on this one could expect the stack to show trashcan functions twice but we only see one. This is what makes me suspect of something in greenlet, because that's hiding the other stack that started the trashcan.

What's unclear here is the exact order of things and the types involved, but that's what's 80% of the debugging time here, unfortunately.

I'm terribly sorry I cannot be more helpful here but I am still ironing out the 3.11 release (and I am on vacation this week) and I don't have time for debugging this at the time :(

Oct 12 '22 15:10 pablogsal

@jamadden Printing aka IO is a bit scary from inside the __free, so I'm seeing if I can use gdb facilities to get it done.

Oct 12 '22 16:10 arcivanov

I've made some advances in traceback/backtrace by reinstalling and recompiled gevent and greenlet from source with CFLAGS='-g -O0' CPPFLAGS='-g -O0'

Here are the results:

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007fba82c8ecb3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007fba82c3e9c6 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007fba82c287f4 in __GI_abort () at abort.c:79
#4  0x00007fba82c2871b in __assert_fail_base (fmt=0x7fba82dbbda0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2274, function=<optimized out>) at assert.c:92
#5  0x00007fba82c37576 in __GI___assert_fail (assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2274, function=0x786cc0 <__PRETTY_FUNCTION__.3> "_PyTrash_thread_destroy_chain") at assert.c:101
#6  0x000000000054985d in _PyTrash_thread_destroy_chain (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2274
#7  0x000000000054992f in _PyTrash_end (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2299
#8  0x000000000055c1ef in tupledealloc (op=0x7fba73de06e0) at Objects/tupleobject.c:215
#9  0x0000000000549ba6 in _Py_Dealloc (
    op=(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, ()), (...)), None), None), None), 14), None), 47), None), None), 50), None), None), None), None), None), 76), None), None), None), 55), None), 67), None), 43), None), 56), None), 63), None), 20), None), None), None), None), None), None), None), None), 73), None), None), None), None), None), 34), None), 54), None), 18), None), 44), None), None), None), None), None), 94), 95)) at Objects/object.c:2387
#10 0x00000000005272d8 in Py_DECREF (filename=0x77e260 "./Include/object.h", lineno=602, 
    op=(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, ()), (...)), None), None), None), 14), None), 47), None), None), 50), None), None), None), None), None), 76), None), None), None), 55), None), 67), None), 43), None), 56), None), 63), None), 20), None), None), None), None), None), None), None), None), 73), None), None), None), None), None), 34), None), 54), None), 18), None), 44), None), None), None), None), None), 94), 95)) at ./Include/object.h:527
#11 0x0000000000527326 in Py_XDECREF (
    op=(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, ()), (...)), None), None), None), 14), None), 47), None), None), 50), None), None), None), None), None), 76), None), None), None), 55), None), 67), None), 43), None), 56), None), 63), None), 20), None), None), None), None), None), None), None), None), 73), None), None), None), None), None), 34), None), 54), None), 18), None), 44), None), None), None), None), None), 94), 95)) at ./Include/object.h:602
#12 0x000000000052b369 in insertdict (mp=0x7fba73debdd0, key='args', hash=-1361140119721033708, value=()) at Objects/dictobject.c:1304
#13 0x000000000052cee8 in _PyDict_SetItem_Take2 (mp=0x7fba73debdd0, key='args', value=()) at Objects/dictobject.c:1886
#14 0x000000000052cfa0 in PyDict_SetItem (
    op={'args': (), 'kwargs': {}, 'value': None, '_start_event': <_dummy_event at remote 0x7fba750b38e0>, '_notifier': <gevent.libev.corecext.callback at remote 0x7fba73bc2210>, '_formatted_info': None, '_links': [], '_ident': None, '_exc_info': (None, None, None), 'spawning_greenlet': None, 'spawn_tree_locals': None, 'spawning_stack': None, '_gevent_local_localimpl_140438784467008': <weakref.ReferenceType at remote 0x7fba73bf1630>}, key='args', value=())
    at Objects/dictobject.c:1906
#15 0x00000000005354c2 in _PyObjectDict_SetItem (tp=0x14209d0, dictptr=0x7fba73d0f2c8, key='args', value=()) at Objects/dictobject.c:5668
#16 0x000000000054890d in _PyObject_GenericSetAttrWithDict (
    obj=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7fba750b38e0>, _notifier=<gevent.libev.corecext.callback at remote 0x7fba73bc2210>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140438784467008=<weakref.ReferenceType at remote 0x7fba73bf1630>) at remote 0x7fba73d0f230>, name='args', value=(), dict=0x0)
    at Objects/object.c:1422
#17 0x0000000000548a54 in PyObject_GenericSetAttr (
    obj=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7fba750b38e0>, _notifier=<gevent.libev.corecext.callback at remote 0x7fba73bc2210>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140438784467008=<weakref.ReferenceType at remote 0x7fba73bf1630>) at remote 0x7fba73d0f230>, name='args', value=())
    at Objects/object.c:1456
#18 0x0000000000547ab4 in PyObject_SetAttr (v=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7fba750b38e0>, _notifier=<gevent.libev.corecext.callback at remote 0x7fba73bc2210>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140438784467008=<weakref.ReferenceType at remote 0x7fba73bf1630>) at remote 0x7fba73d0f230>, 
    name='args', value=()) at Objects/object.c:1028
#19 0x0000000000610597 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba7392c0a0, throwflag=0) at Python/ceval.c:2889
#20 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba7392c020, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#21 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba750c03c0, locals=0x0, args=0x7ffd77752d20, argcount=1, kwnames=0x0) at Python/ceval.c:6428
#22 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba750c03c0>, stack=0x7ffd77752d20, nargsf=1, kwnames=0x0) at Objects/call.c:393
#23 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba750c03c0>, args=0x7ffd77752d20, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#24 0x00000000004e09e6 in method_vectorcall (method=<method at remote 0x7fba73bbee70>, args=0x9dda90 <_PyRuntime+58928>, nargsf=0, kwnames=0x0) at Objects/classobject.c:67
#25 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fba73bbee70>, tuple=(), kwargs=0x0) at Objects/call.c:245
#26 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fba73bbee70>, args=(), kwargs=0x0) at Objects/call.c:328
#27 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fba73bbee70>, args=(), kwargs=0x0) at Objects/call.c:355
#28 0x00007fba750e0d02 in g_initialstub (mark=0x7ffd77752f78) at src/greenlet/greenlet.c:960
#29 0x00007fba750e047e in g_switch (target=0x7fba73d0f230, args=(), kwargs=0x0) at src/greenlet/greenlet.c:716
#30 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba7506e570>, stack=0x7ffd777530c0, nargsf=2, kwnames=0x0) at Objects/call.c:393
#31 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba7506e570>, args=0x7ffd777530c0, nargsf=2, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#32 0x00000000004e0ab1 in method_vectorcall (method=<method at remote 0x7fba73829d90>, args=0x7fba73ded8c8, nargsf=1, kwnames=0x0) at Objects/classobject.c:89
#33 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fba73829d90>, tuple=(<Waiter at remote 0x7fba73829b50>,), kwargs=0x0) at Objects/call.c:245
#34 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fba73829d90>, args=(<Waiter at remote 0x7fba73829b50>,), kwargs=0x0) at Objects/call.c:328
#35 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fba73829d90>, args=(<Waiter at remote 0x7fba73829b50>,), kwargs=0x0) at Objects/call.c:355
#36 0x00007fba750f655d in gevent_callback (loop=0x7fba7453cf50, callback=<method at remote 0x7fba73829d90>, args=(<Waiter at remote 0x7fba73829b50>,), watcher=<gevent.libev.corecext.io at remote 0x7fba73ba5c70>, c_watcher=0x7fba73ba5cb0, revents=1) at src/gevent/libev/callbacks.c:106
#37 0x00007fba750f68a1 in gevent_callback_io (_loop=0x7fba75160da0 <default_loop_struct>, c_watcher=0x7fba73ba5cb0, revents=1) at src/gevent/libev/callbacks.c:187
#38 0x00007fba750fd18b in ev_invoke_pending (loop=0x7fba75160da0 <default_loop_struct>) at /tmp/pip-install-18egfds3/gevent_0a60da4b6e50406b82e7e73be8ea01a3/deps/libev/ev.c:3770
#39 0x00007fba750fdc6d in ev_run (loop=0x7fba75160da0 <default_loop_struct>, flags=0) at /tmp/pip-install-18egfds3/gevent_0a60da4b6e50406b82e7e73be8ea01a3/deps/libev/ev.c:4190
#40 0x00007fba7510eeb8 in __pyx_pf_6gevent_5libev_8corecext_4loop_14run (__pyx_v_self=0x7fba7453cf50, __pyx_v_nowait=False, __pyx_v_once=False) at src/gevent/libev/corecext.c:9820
#41 0x00007fba7510ed22 in __pyx_pw_6gevent_5libev_8corecext_4loop_15run (__pyx_v_self=<gevent.libev.corecext.loop at remote 0x7fba7453cf50>, __pyx_args=0x7fba74ea9090, __pyx_nargs=0, __pyx_kwds=0x0) at src/gevent/libev/corecext.c:9776
#42 0x00007fba75142fee in __Pyx_CyFunction_Vectorcall_FASTCALL_KEYWORDS (func=<_cython_3_0_0a11.cython_function_or_method at remote 0x7fba752abc50>, args=0x7fba74ea9090, nargsf=9223372036854775809, kwnames=0x0) at src/gevent/libev/corecext.c:27668
#43 0x00000000004dc379 in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<_cython_3_0_0a11.cython_function_or_method at remote 0x7fba752abc50>, args=0x7fba74ea9088, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#44 0x00000000004dcf80 in PyObject_Vectorcall (callable=<_cython_3_0_0a11.cython_function_or_method at remote 0x7fba752abc50>, args=0x7fba74ea9088, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#45 0x000000000061b927 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba74ea9020, throwflag=0) at Python/ceval.c:4772
#46 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba74ea9020, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#47 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba750c2db0, locals=0x0, args=0x7ffd77755060, argcount=1, kwnames=0x0) at Python/ceval.c:6428
#48 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba750c2db0>, stack=0x7ffd77755060, nargsf=1, kwnames=0x0) at Objects/call.c:393
#49 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba750c2db0>, args=0x7ffd77755060, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#50 0x00000000004e09e6 in method_vectorcall (method=<method at remote 0x7fba74837f50>, args=0x9dda90 <_PyRuntime+58928>, nargsf=0, kwnames=0x0) at Objects/classobject.c:67
#51 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fba74837f50>, tuple=(), kwargs=0x0) at Objects/call.c:245
#52 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fba74837f50>, args=(), kwargs=0x0) at Objects/call.c:328
#53 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fba74837f50>, args=(), kwargs=0x0) at Objects/call.c:355
#54 0x00007fba750e0d02 in g_initialstub (mark=0x7ffd777552b8) at src/greenlet/greenlet.c:960
#55 0x00007fba750e047e in g_switch (target=0x7fba73d0f5b0, args=(), kwargs=0x0) at src/greenlet/greenlet.c:716
#56 0x00007fba750e198c in green_switch (self=0x7fba73d0f5b0, args=(), kwargs=0x0) at src/greenlet/greenlet.c:1356
#57 0x00000000004eca75 in method_vectorcall_VARARGS_KEYWORDS (func=<method_descriptor at remote 0x7fba752f0350>, args=0x7fba82f02fa8, nargsf=9223372036854775809, kwnames=0x0) at Objects/descrobject.c:364
#58 0x00000000004dc379 in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method_descriptor at remote 0x7fba752f0350>, args=0x7fba82f02fa8, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#59 0x00000000004dcf80 in PyObject_Vectorcall (callable=<method_descriptor at remote 0x7fba752f0350>, args=0x7fba82f02fa8, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#60 0x000000000061b927 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02f40, throwflag=0) at Python/ceval.c:4772
#61 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02c80, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#62 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba73dbf540, locals=0x0, args=0x7fba73db0608, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#63 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba73dbf540>, stack=0x7fba73db0608, nargsf=2, kwnames=0x0) at Objects/call.c:393
#64 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4dd24a <_PyFunction_Vectorcall>, callable=<function at remote 0x7fba73dbf540>, tuple=('/tmp/tmppl37e7y3', False), kwargs={}) at Objects/call.c:245
#65 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba73dbf540>, args=('/tmp/tmppl37e7y3', False), kwargs={}) at Objects/call.c:328
#66 0x00000000004dd193 in PyObject_Call (callable=<function at remote 0x7fba73dbf540>, args=('/tmp/tmppl37e7y3', False), kwargs={}) at Objects/call.c:355
#67 0x0000000000625967 in do_call_core (tstate=0x9f7e18 <_PyRuntime+166328>, func=<function at remote 0x7fba73dbf540>, callargs=('/tmp/tmppl37e7y3', False), kwdict={}, use_tracing=0) at Python/ceval.c:7347
#68 0x000000000061f5d4 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02c08, throwflag=0) at Python/ceval.c:5377
#69 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f029a8, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#70 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba744f2570, locals=0x0, args=0x7ffd77758b30, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#71 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba744f2570>, stack=0x7ffd77758b30, nargsf=2, kwnames=0x0) at Objects/call.c:393
#72 0x00000000004dc8dd in _PyObject_FastCallDictTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba744f2570>, args=0x7ffd77758b30, nargsf=2, kwargs=0x0) at Objects/call.c:141
#73 0x00000000004dd66b in _PyObject_Call_Prepend (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba744f2570>, obj=<Popen(returncode=None, finalizer=None, sentinel=None, pid=0) at remote 0x7fba73dd1690>, 
    args=(<ForkProcess(_identity=(1,), _config={'authkey': <AuthenticationString at remote 0x7fba7537e0d0>, 'semprefix': '/mp'}, _parent_pid=392823, _parent_name='MainProcess', _popen=None, _closed=False, _target=<function at remote 0x7fba73dbf540>, _args=('/tmp/tmppl37e7y3', False), _kwargs={}, _name='ForkProcess-1') at remote 0x7fba73dd15a0>,), kwargs=0x0) at Objects/call.c:482
#74 0x000000000057337c in slot_tp_init (self=<Popen(returncode=None, finalizer=None, sentinel=None, pid=0) at remote 0x7fba73dd1690>, 
    args=(<ForkProcess(_identity=(1,), _config={'authkey': <AuthenticationString at remote 0x7fba7537e0d0>, 'semprefix': '/mp'}, _parent_pid=392823, _parent_name='MainProcess', _popen=None, _closed=False, _target=<function at remote 0x7fba73dbf540>, _args=('/tmp/tmppl37e7y3', False), _kwargs={}, _name='ForkProcess-1') at remote 0x7fba73dd15a0>,), kwds=0x0) at Objects/typeobject.c:7861
#75 0x0000000000561adb in type_call (type=0x17c6ae0, args=(<ForkProcess(_identity=(1,), _config={'authkey': <AuthenticationString at remote 0x7fba7537e0d0>, 'semprefix': '/mp'}, _parent_pid=392823, _parent_name='MainProcess', _popen=None, _closed=False, _target=<function at remote 0x7fba73dbf540>, _args=('/tmp/tmppl37e7y3', False), _kwargs={}, _name='ForkProcess-1') at remote 0x7fba73dd15a0>,), kwds=0x0) at Objects/typeobject.c:1112
#76 0x00000000004dcbee in _PyObject_MakeTpCall (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<type at remote 0x17c6ae0>, args=0x7fba82f029a0, nargs=1, keywords=0x0) at Objects/call.c:214
#77 0x00000000004dc35d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<type at remote 0x17c6ae0>, args=0x7fba82f029a0, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#78 0x00000000004dcf80 in PyObject_Vectorcall (callable=<type at remote 0x17c6ae0>, args=0x7fba82f029a0, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#79 0x000000000061b927 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02938, throwflag=0) at Python/ceval.c:4772
#80 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f025f0, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#81 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba7440d440, locals=0x0, args=0x7ffd7775a810, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#82 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba7440d440>, stack=0x7ffd7775a810, nargsf=2, kwnames=0x0) at Objects/call.c:393
#83 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba7440d440>, args=0x7ffd7775a810, nargsf=2, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#84 0x00000000004e0ab1 in method_vectorcall (method=<method at remote 0x7fba745948f0>, args=0x7fba73d1ed68, nargsf=1, kwnames=0x0) at Objects/classobject.c:89
#85 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fba745948f0>, 
    tuple=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs={}) at Objects/call.c:245
#86 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fba745948f0>, 
    args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs={}) at Objects/call.c:328
#87 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fba745948f0>, 
    args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs={}) at Objects/call.c:355
#88 0x0000000000625967 in do_call_core (tstate=0x9f7e18 <_PyRuntime+166328>, func=<method at remote 0x7fba745948f0>, 
    callargs=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwdict={}, use_tracing=0) at Python/ceval.c:7347
#89 0x000000000061f5d4 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02568, throwflag=0) at Python/ceval.c:5377
#90 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02568, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#91 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba7440d650, locals=0x0, args=0x7ffd7775c510, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#92 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba7440d650>, stack=0x7ffd7775c510, nargsf=2, kwnames=0x0) at Objects/call.c:393
#93 0x00000000004dc8dd in _PyObject_FastCallDictTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba7440d650>, args=0x7ffd7775c510, nargsf=2, kwargs=0x0) at Objects/call.c:141
#94 0x00000000004dd66b in _PyObject_Call_Prepend (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba7440d650>, 
    obj=<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertListEqual', <type at r...(truncated), 
    args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs=0x0) at Objects/call.c:482
#95 0x0000000000572cbf in slot_tp_call (
    self=<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False,--Type <RET> for more, q to quit, c to continue without paging--c
 stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertListEqual', <type at r...(truncated), args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwds=0x0) at Objects/typeobject.c:7630
#96 0x00000000004dcbee in _PyObject_MakeTpCall (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertListEqual', <type at r...(truncated), args=0x7fba82f02550, nargs=1, keywords=0x0) at Objects/call.c:214
#97 0x00000000004dc35d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertListEqual', <type at r...(truncated), args=0x7fba82f02550, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#98 0x00000000004dcf80 in PyObject_Vectorcall (callable=<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertListEqual', <type at r...(truncated), args=0x7fba82f02550, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#99 0x000000000061b927 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f024c0, throwflag=0) at Python/ceval.c:4772
#100 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f024c0, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#101 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba744112e0, locals=0x0, args=0x7ffd7775e1a0, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#102 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba744112e0>, stack=0x7ffd7775e1a0, nargsf=2, kwnames=0x0) at Objects/call.c:393
#103 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba744112e0>, args=0x7ffd7775e1a0, nargsf=2, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#104 0x00000000004e0ab1 in method_vectorcall (method=<method at remote 0x7fba74662750>, args=0x7fba73d1ebd8, nargsf=1, kwnames=0x0) at Objects/classobject.c:89
#105 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fba74662750>, tuple=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs={}) at Objects/call.c:245
#106 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fba74662750>, args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs={}) at Objects/call.c:328
#107 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fba74662750>, args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs={}) at Objects/call.c:355
#108 0x0000000000625967 in do_call_core (tstate=0x9f7e18 <_PyRuntime+166328>, func=<method at remote 0x7fba74662750>, callargs=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwdict={}, use_tracing=0) at Python/ceval.c:7347
#109 0x000000000061f5d4 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02438, throwflag=0) at Python/ceval.c:5377
#110 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02438, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#111 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba74411180, locals=0x0, args=0x7ffd7775fea0, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#112 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba74411180>, stack=0x7ffd7775fea0, nargsf=2, kwnames=0x0) at Objects/call.c:393
#113 0x00000000004dc8dd in _PyObject_FastCallDictTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba74411180>, args=0x7ffd7775fea0, nargsf=2, kwargs=0x0) at Objects/call.c:141
#114 0x00000000004dd66b in _PyObject_Call_Prepend (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba74411180>, obj=<TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertLis...(truncated), args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs=0x0) at Objects/call.c:482
#115 0x0000000000572cbf in slot_tp_call (self=<TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertLis...(truncated), args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwds=0x0) at Objects/typeobject.c:7630
#116 0x00000000004dcbee in _PyObject_MakeTpCall (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertLis...(truncated), args=0x7fba82f02420, nargs=1, keywords=0x0) at Objects/call.c:214
#117 0x00000000004dc35d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertLis...(truncated), args=0x7fba82f02420, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#118 0x00000000004dcf80 in PyObject_Vectorcall (callable=<TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertLis...(truncated), args=0x7fba82f02420, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#119 0x000000000061b927 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02390, throwflag=0) at Python/ceval.c:4772
#120 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02390, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#121 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba744112e0, locals=0x0, args=0x7ffd77761b30, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#122 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba744112e0>, stack=0x7ffd77761b30, nargsf=2, kwnames=0x0) at Objects/call.c:393
#123 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba744112e0>, args=0x7ffd77761b30, nargsf=2, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#124 0x00000000004e0ab1 in method_vectorcall (method=<method at remote 0x7fba7465af90>, args=0x7fba73d1eb88, nargsf=1, kwnames=0x0) at Objects/classobject.c:89
#125 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fba7465af90>, tuple=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs={}) at Objects/call.c:245
#126 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fba7465af90>, args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs={}) at Objects/call.c:328
#127 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fba7465af90>, args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs={}) at Objects/call.c:355
#128 0x0000000000625967 in do_call_core (tstate=0x9f7e18 <_PyRuntime+166328>, func=<method at remote 0x7fba7465af90>, callargs=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwdict={}, use_tracing=0) at Python/ceval.c:7347
#129 0x000000000061f5d4 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02308, throwflag=0) at Python/ceval.c:5377
#130 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02308, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#131 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba74411180, locals=0x0, args=0x7ffd77763830, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#132 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba74411180>, stack=0x7ffd77763830, nargsf=2, kwnames=0x0) at Objects/call.c:393
#133 0x00000000004dc8dd in _PyObject_FastCallDictTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba74411180>, args=0x7ffd77763830, nargsf=2, kwargs=0x0) at Objects/call.c:141
#134 0x00000000004dd66b in _PyObject_Call_Prepend (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba74411180>, obj=<TestSuite(_tests=[None, <TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at rem...(truncated), args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwargs=0x0) at Objects/call.c:482
#135 0x0000000000572cbf in slot_tp_call (self=<TestSuite(_tests=[None, <TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at rem...(truncated), args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>,), kwds=0x0) at Objects/typeobject.c:7630
#136 0x00000000004dcbee in _PyObject_MakeTpCall (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSuite(_tests=[None, <TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at rem...(truncated), args=0x7fba82f022e0, nargs=1, keywords=0x0) at Objects/call.c:214
#137 0x00000000004dc35d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSuite(_tests=[None, <TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at rem...(truncated), args=0x7fba82f022e0, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#138 0x00000000004dcf80 in PyObject_Vectorcall (callable=<TestSuite(_tests=[None, <TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fba75406210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fba75406300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fba75406300>) at remote 0x7fba73dd1050>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x18c7c00>) at remote 0x7fba744b9fa0>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fba74a65a50>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at rem...(truncated), args=0x7fba82f022e0, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#139 0x000000000061b927 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02200, throwflag=0) at Python/ceval.c:4772
#140 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02090, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#141 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba73d2d230, locals=0x0, args=0x7ffd777654e0, argcount=1, kwnames=0x0) at Python/ceval.c:6428
#142 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fba73d2d230>, stack=0x7ffd777654e0, nargsf=1, kwnames=0x0) at Objects/call.c:393
#143 0x00000000004dc8dd in _PyObject_FastCallDictTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba73d2d230>, args=0x7ffd777654e0, nargsf=1, kwargs=0x0) at Objects/call.c:141
#144 0x00000000004dd66b in _PyObject_Call_Prepend (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fba73d2d230>, obj=<TestProgram(module=<module at remote 0x7fba7542fef0>, exit=True, failfast=False, catchbreak=False, verbosity=1, buffer=False, tb_locals=False, warnings='default', defaultTest=None, testRunner=<type at remote 0x18811a0>, testLoader=<TestLoader(errors=[], _loading_packages=set()) at remote 0x7fba7440acc0>, progName='monkey_manager_tests.py', testNamePatterns=[], _main_parser=<ArgumentParser(description=None, argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x18767d0>, 'store': <type at remote 0x18767d0>, 'store_const': <type at remote 0x1876f10>, 'store_true': <type at remote 0x1877650>, 'store_false': <type at remote 0x1877d90>, 'append': <type at remote 0x18784d0>, 'append_const': <type at remote 0x1878c10>, 'count': <type at remote 0x1879350>, 'help': <type at remote 0x1879a90>, 'version': <type at remote 0x187a800>, 'parsers': <type at remote 0x187b340>, 'extend': <type at remote 0x187ba80>}, 'type': {None: <function at remote 0x7fba73dbfac0>}...(truncated), args=(), kwargs=0x0) at Objects/call.c:482
#145 0x000000000057337c in slot_tp_init (self=<TestProgram(module=<module at remote 0x7fba7542fef0>, exit=True, failfast=False, catchbreak=False, verbosity=1, buffer=False, tb_locals=False, warnings='default', defaultTest=None, testRunner=<type at remote 0x18811a0>, testLoader=<TestLoader(errors=[], _loading_packages=set()) at remote 0x7fba7440acc0>, progName='monkey_manager_tests.py', testNamePatterns=[], _main_parser=<ArgumentParser(description=None, argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x18767d0>, 'store': <type at remote 0x18767d0>, 'store_const': <type at remote 0x1876f10>, 'store_true': <type at remote 0x1877650>, 'store_false': <type at remote 0x1877d90>, 'append': <type at remote 0x18784d0>, 'append_const': <type at remote 0x1878c10>, 'count': <type at remote 0x1879350>, 'help': <type at remote 0x1879a90>, 'version': <type at remote 0x187a800>, 'parsers': <type at remote 0x187b340>, 'extend': <type at remote 0x187ba80>}, 'type': {None: <function at remote 0x7fba73dbfac0>}...(truncated), args=(), kwds=0x0) at Objects/typeobject.c:7861
#146 0x0000000000561adb in type_call (type=0x1881f60, args=(), kwds=0x0) at Objects/typeobject.c:1112
#147 0x00000000004dcbee in _PyObject_MakeTpCall (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<type at remote 0x1881f60>, args=0x7fba82f02078, nargs=0, keywords=0x0) at Objects/call.c:214
#148 0x00000000004dc35d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<type at remote 0x1881f60>, args=0x7fba82f02078, nargsf=9223372036854775808, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#149 0x00000000004dcf80 in PyObject_Vectorcall (callable=<type at remote 0x1881f60>, args=0x7fba82f02078, nargsf=9223372036854775808, kwnames=0x0) at Objects/call.c:299
#150 0x000000000061b927 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02020, throwflag=0) at Python/ceval.c:4772
#151 0x0000000000605ff0 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fba82f02020, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#152 0x0000000000623424 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fba754acc00, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fba75431c80>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fba7559f6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fba75367750>, 'monkey': <module at remote 0x7fba75340a70>, 'TestCase': <type at remote 0x1851660>, 'main': <type at remote 0x1881f60>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fba74664b90>, 'spawn': <method at remote 0x7fba752c2390>, 'wait': <function at remote 0x7fba750a8260>, 'get_hub': <function at remote 0x7fba7506d390>, 'sleep': <function at remote 0x7fba750c1440>, 'mp': <module at remote 0x7fba75340b30>, 'mktemp': <function at remote 0x...(truncated), args=0x0, argcount=0, kwnames=0x0) at Python/ceval.c:6428
#153 0x00000000006090e3 in PyEval_EvalCode (co=<code at remote 0x1334be0>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fba75431c80>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fba7559f6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fba75367750>, 'monkey': <module at remote 0x7fba75340a70>, 'TestCase': <type at remote 0x1851660>, 'main': <type at remote 0x1881f60>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fba74664b90>, 'spawn': <method at remote 0x7fba752c2390>, 'wait': <function at remote 0x7fba750a8260>, 'get_hub': <function at remote 0x7fba7506d390>, 'sleep': <function at remote 0x7fba750c1440>, 'mp': <module at remote 0x7fba75340b30>, 'mktemp': <function at remote 0x...(truncated), locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fba75431c80>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fba7559f6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fba75367750>, 'monkey': <module at remote 0x7fba75340a70>, 'TestCase': <type at remote 0x1851660>, 'main': <type at remote 0x1881f60>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fba74664b90>, 'spawn': <method at remote 0x7fba752c2390>, 'wait': <function at remote 0x7fba750a8260>, 'get_hub': <function at remote 0x7fba7506d390>, 'sleep': <function at remote 0x7fba750c1440>, 'mp': <module at remote 0x7fba75340b30>, 'mktemp': <function at remote 0x...(truncated)) at Python/ceval.c:1154
#154 0x0000000000686e3a in run_eval_code_obj (tstate=0x9f7e18 <_PyRuntime+166328>, co=0x1334be0, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fba75431c80>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fba7559f6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fba75367750>, 'monkey': <module at remote 0x7fba75340a70>, 'TestCase': <type at remote 0x1851660>, 'main': <type at remote 0x1881f60>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fba74664b90>, 'spawn': <method at remote 0x7fba752c2390>, 'wait': <function at remote 0x7fba750a8260>, 'get_hub': <function at remote 0x7fba7506d390>, 'sleep': <function at remote 0x7fba750c1440>, 'mp': <module at remote 0x7fba75340b30>, 'mktemp': <function at remote 0x...(truncated), locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fba75431c80>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fba7559f6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fba75367750>, 'monkey': <module at remote 0x7fba75340a70>, 'TestCase': <type at remote 0x1851660>, 'main': <type at remote 0x1881f60>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fba74664b90>, 'spawn': <method at remote 0x7fba752c2390>, 'wait': <function at remote 0x7fba750a8260>, 'get_hub': <function at remote 0x7fba7506d390>, 'sleep': <function at remote 0x7fba750c1440>, 'mp': <module at remote 0x7fba75340b30>, 'mktemp': <function at remote 0x...(truncated)) at Python/pythonrun.c:1714
#155 0x0000000000686f26 in run_mod (mod=0x13492f0, filename='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fba75431c80>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fba7559f6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fba75367750>, 'monkey': <module at remote 0x7fba75340a70>, 'TestCase': <type at remote 0x1851660>, 'main': <type at remote 0x1881f60>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fba74664b90>, 'spawn': <method at remote 0x7fba752c2390>, 'wait': <function at remote 0x7fba750a8260>, 'get_hub': <function at remote 0x7fba7506d390>, 'sleep': <function at remote 0x7fba750c1440>, 'mp': <module at remote 0x7fba75340b30>, 'mktemp': <function at remote 0x...(truncated), locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fba75431c80>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fba7559f6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fba75367750>, 'monkey': <module at remote 0x7fba75340a70>, 'TestCase': <type at remote 0x1851660>, 'main': <type at remote 0x1881f60>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fba74664b90>, 'spawn': <method at remote 0x7fba752c2390>, 'wait': <function at remote 0x7fba750a8260>, 'get_hub': <function at remote 0x7fba7506d390>, 'sleep': <function at remote 0x7fba750c1440>, 'mp': <module at remote 0x7fba75340b30>, 'mktemp': <function at remote 0x...(truncated), flags=0x7ffd77767368, arena=0x7fba75322e00) at Python/pythonrun.c:1735
#156 0x0000000000686c16 in pyrun_file (fp=0x131d140, filename='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', start=257, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fba75431c80>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fba7559f6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fba75367750>, 'monkey': <module at remote 0x7fba75340a70>, 'TestCase': <type at remote 0x1851660>, 'main': <type at remote 0x1881f60>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fba74664b90>, 'spawn': <method at remote 0x7fba752c2390>, 'wait': <function at remote 0x7fba750a8260>, 'get_hub': <function at remote 0x7fba7506d390>, 'sleep': <function at remote 0x7fba750c1440>, 'mp': <module at remote 0x7fba75340b30>, 'mktemp': <function at remote 0x...(truncated), locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fba75431c80>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fba7559f6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fba75367750>, 'monkey': <module at remote 0x7fba75340a70>, 'TestCase': <type at remote 0x1851660>, 'main': <type at remote 0x1881f60>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fba74664b90>, 'spawn': <method at remote 0x7fba752c2390>, 'wait': <function at remote 0x7fba750a8260>, 'get_hub': <function at remote 0x7fba7506d390>, 'sleep': <function at remote 0x7fba750c1440>, 'mp': <module at remote 0x7fba75340b30>, 'mktemp': <function at remote 0x...(truncated), closeit=1, flags=0x7ffd77767368) at Python/pythonrun.c:1630
#157 0x00000000006844d3 in _PyRun_SimpleFileObject (fp=0x131d140, filename='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', closeit=1, flags=0x7ffd77767368) at Python/pythonrun.c:440
#158 0x0000000000683a30 in _PyRun_AnyFileObject (fp=0x131d140, filename='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', closeit=1, flags=0x7ffd77767368) at Python/pythonrun.c:79
#159 0x00000000006afe2f in pymain_run_file_obj (program_name='/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/bin/python3.11', filename='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', skip_source_first_line=0) at Modules/main.c:360
#160 0x00000000006afee6 in pymain_run_file (config=0x9dde60 <_PyRuntime+59904>) at Modules/main.c:379
#161 0x00000000006b0515 in pymain_run_python (exitcode=0x7ffd777674ec) at Modules/main.c:601
#162 0x00000000006b0618 in Py_RunMain () at Modules/main.c:680
#163 0x00000000006b06ac in pymain_main (args=0x7ffd77767550) at Modules/main.c:710
#164 0x00000000006b0726 in Py_BytesMain (argc=2, argv=0x7ffd777676a8) at Modules/main.c:734
#165 0x000000000041d7e6 in main (argc=2, argv=0x7ffd777676a8) at ./Programs/python.c:15
(gdb) py-bt
Traceback (most recent call first):
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/greenlet.py", line 925, in __free
    self.args = ()
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/greenlet.py", line 914, in run
    self.__free()
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/hub.py", line 647, in run
    loop.run()
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/_greenlet_primitives.py", line 65, in switch
    return _greenlet_switch(self) # pylint:disable=undefined-variable
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/_waiter.py", line 154, in get
    return self.hub.switch()
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/hub.py", line 742, in join
    waiter.get()
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/_hub_primitives.py", line 286, in wait_on_objects
    return hub.join(timeout=timeout) # pylint:disable=
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/_mp_manager_server.py", line 130, in _manager_process
    wait(timeout=300)
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/_mp_manager_server.py", line 90, in manager_process
    _manager_process(addr)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/popen_fork.py", line 71, in _launch
    code = process_obj._bootstrap(parent_sentinel=child_r)
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/geventmp/_mp/3/_mp_popen_fork.py", line 27, in _launch
    super()._launch(process_obj)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/context.py", line 281, in _Popen
    return Popen(process_obj)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 85, in _test_manager
    p.start()
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 81, in run_manager_test
    self._test_manager(ctx, remote_trace)
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 71, in test_manager_fork
    self.run_manager_test("fork")
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/case.py", line 623, in run
    self._callTestMethod(testMethod)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/case.py", line 678, in __call__
    return self.run(*args, **kwds)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/suite.py", line 122, in run
    test(result)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/suite.py", line 122, in run
    test(result)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/runner.py", line 217, in run
    test(result)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/main.py", line 274, in runTests
    self.result = testRunner.run(self.test)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/main.py", line 102, in __init__
    self.runTests()
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 130, in <module>
    main()
(gdb) py-locals
Locals for __free
self = <Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7fba750b38e0>, _notifier=<gevent.libev.corecext.callback at remote 0x7fba73bc2210>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140438784467008=<weakref.ReferenceType at remote 0x7fba73bf1630>) at remote 0x7fba73d0f230>
Locals for run
self = <Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7fba750b38e0>, _notifier=<gevent.libev.corecext.callback at remote 0x7fba73bc2210>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140438784467008=<weakref.ReferenceType at remote 0x7fba73bf1630>) at remote 0x7fba73d0f230>
result = None

Oct 12 '22 16:10 arcivanov

core.python3\x2e11.1000.a73c765f59ff4f388aea5262a29a16b9.392825.1665590767000000.gz

Oct 12 '22 16:10 arcivanov

Oooffff... Printing the self.args makes the assertion go away.

    def __free(self):
        try:
            # It seems that Cython 0.29.13 sometimes miscompiles
            # self.__dict__.pop('_run', None) ? When we moved this out of the
            # inline finally: block in run(), we started getting strange
            # exceptions from places that subclassed Greenlet.
            del self._run
        except AttributeError:
            pass

        print(repr(self.args), file=stderr, flush=True)
        self.args = ()
        self.kwargs.clear()

test.err.log.gz

Oct 12 '22 16:10 arcivanov

Looks like it's some weirdness in a traceback object!

Python built with this patch:

$ cat object.patch 
--- Objects/object.c    2022-10-12 17:34:12.591666507 -0400
+++ Objects/object.new.c        2022-10-12 17:41:53.125468991 -0400
@@ -2229,9 +2229,8 @@
  * object, with refcount 0.  Py_DECREF must already have been called on it.
  */
 static void
-_PyTrash_thread_deposit_object(PyObject *op)
+_PyTrash_thread_deposit_object(PyThreadState *tstate, PyObject *op)
 {
-    PyThreadState *tstate = _PyThreadState_GET();
     _PyObject_ASSERT(op, _PyObject_IS_GC(op));
     _PyObject_ASSERT(op, !_PyObject_GC_IS_TRACKED(op));
     _PyObject_ASSERT(op, Py_REFCNT(op) == 0);
@@ -2242,9 +2241,8 @@
 /* Deallocate all the objects in the gcstate->trash_delete_later list.
  * Called when the call-stack unwinds again. */
 static void
-_PyTrash_thread_destroy_chain(void)
+_PyTrash_thread_destroy_chain(PyThreadState *tstate)
 {
-    PyThreadState *tstate = _PyThreadState_GET();
     /* We need to increase trash_delete_nesting here, otherwise,
        _PyTrash_thread_destroy_chain will be called recursively
        and then possibly crash.  An example that may crash without
@@ -2272,6 +2270,14 @@
          * up distorting allocation statistics.
          */
         _PyObject_ASSERT(op, Py_REFCNT(op) == 0);
+        PyTypeObject *type = Py_TYPE(op);
+        if(type && strcmp(type->tp_name, "frame")) {
+          op->ob_refcnt++;
+        }
+        _PyObject_Dump(op);
+        if(type && strcmp(type->tp_name, "frame")) {
+          op->ob_refcnt--;
+        }
         (*dealloc)(op);
         assert(tstate->trash_delete_nesting == 1);
     }
@@ -2285,7 +2291,7 @@
     if (tstate->trash_delete_nesting >= _PyTrash_UNWIND_LEVEL) {
         /* Store the object (to be deallocated later) and jump past
          * Py_TRASHCAN_END, skipping the body of the deallocator */
-        _PyTrash_thread_deposit_object(op);
+        _PyTrash_thread_deposit_object(tstate, op);
         return 1;
     }
     ++tstate->trash_delete_nesting;
@@ -2298,7 +2304,7 @@
 {
     --tstate->trash_delete_nesting;
     if (tstate->trash_delete_later && tstate->trash_delete_nesting <= 0) {
-        _PyTrash_thread_destroy_chain();
+        _PyTrash_thread_destroy_chain(tstate);
     }
 }

Run as follows: PURE_PYTHON=1 GEVENT_TRACK_GREENLET_TREE=0 '/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/bin/python3.11' '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py'

Got this output:

...
object address  : 0x7f422cb741d0
object refcount : 1
object type     : 0x8ddac0
object type name: tuple
object repr     : (<loop at 0x7f422d7c4f50 epoll default pending=0 ref=36 fileno=3>, True, None)
object address  : 0x7f422cb74170
object refcount : 1
object type     : 0x8d76c0
object type name: dict
object repr     : {}
object address  : 0x7f422ce32ed0
object refcount : 1
object type     : 0x8d76c0
object type name: dict
object repr     : {}
object address  : 0x7f422cabaad0
object refcount : 1
object type     : 0x8ddac0
object type name: tuple
object repr     : ('Request #89, Value #73',)
object address  : 0x7f422cb78230
object refcount : 1
object type     : 0x8ddac0
object type name: tuple
object repr     : ('#RETURN', None)
object address  : 0x7f422cb78140
object refcount : 1
object type     : 0x8ddac0
object type name: tuple
object repr     : (25,)
object address  : 0x7f422cac8f50
object refcount : 0
object type     : 0x8d4e40
object type name: frame
object repr     : <refcnt 0 at 0x7f422cac8f50>
object address  : 0x7f422ceaf4d0
object refcount : 0
object type     : 0x8d4e40
object type name: frame
object repr     : <refcnt 0 at 0x7f422ceaf4d0>
object address  : 0x7f422ceb1fe0
object refcount : 1
object type     : 0x8ddac0
object type name: tuple
object repr     : (<gevent._waiter.Waiter object at 0x7f422ce9d370>,)
object address  : 0x7f422cacd190
object refcount : 1
object type     : 0x8da440
object type name: builtin_function_or_method
object repr     : <built-in method switch of Greenlet object at 0x7f422cfd33f0>
object address  : 0x7f422cb706e0
object refcount : 0
object type     : 0x8d4e40
object type name: frame
object repr     : <refcnt 0 at 0x7f422cb706e0>
object address  : 0x7f422cfcfb60
object refcount : 1
object type     : 0x8ddac0
object type name: tuple
object repr     : (<gevent._waiter.Waiter object at 0x7f422ce9d370>,)
object address  : 0x7f422cb67fc0
object refcount : 1
object type     : 0x8ddac0
object type name: tuple
object repr     : (<Hub '' at 0x7f422cf0b5b0 epoll default pending=0 ref=36 fileno=3 thread_ident=0x7f423c0ad740>,)
object address  : 0x7f422cb689f0
object refcount : 1
object type     : 0x8ddac0
object type name: tuple
object repr     : (<loop at 0x7f422d7c4f50 epoll default pending=0 ref=36 fileno=3>, 36, 1, True, None)
object address  : 0x7f422cb63f50
object refcount : 1
object type     : 0x8ddac0
object type name: tuple
object repr     : (<io at 0x7f422ce22600 native=0x7f422ce22640 fd=36 events=READ>, <loop at 0x7f422d7c4f50 epoll default pending=0 ref=36 fileno=3>, True, None)
object address  : 0x7f422cb63fb0
object refcount : 1
object type     : 0x8ddac0
object type name: tuple
object repr     : (<loop at 0x7f422d7c4f50 epoll default pending=0 ref=36 fileno=3>, True, None)
object address  : 0x7f422cb67f70
object refcount : 1
object type     : 0x8e7500
object type name: traceback
object repr     : <traceback object at 0x7f422cb67f70>
python3.11: Objects/object.c:2282: _PyTrash_thread_destroy_chain: Assertion `tstate->trash_delete_nesting == 1' failed.

With this GDB:

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007fa49c08ecb3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007fa49c03e9c6 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007fa49c0287f4 in __GI_abort () at abort.c:79
#4  0x00007fa49c02871b in __assert_fail_base (fmt=0x7fa49c1bbda0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2282, 
    function=<optimized out>) at assert.c:92
#5  0x00007fa49c037576 in __GI___assert_fail (assertion=0x786770 "tstate->trash_delete_nesting == 1", file=0x785d1c "Objects/object.c", line=2282, function=0x786cc0 <__PRETTY_FUNCTION__.3> "_PyTrash_thread_destroy_chain")
    at assert.c:101
#6  0x00000000005498dd in _PyTrash_thread_destroy_chain (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2282
#7  0x00000000005499af in _PyTrash_end (tstate=0x9f7e18 <_PyRuntime+166328>) at Objects/object.c:2307
#8  0x000000000055c26f in tupledealloc (op=0x7fa48d0d2ee0) at Objects/tupleobject.c:215
#9  0x0000000000549c26 in _Py_Dealloc (
    op=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), 19), None), None), None), None), None), None), None), 24), None), None), None), None), None), None), None), None), 63), None), ()), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), 32), None), None), None), None), None), None), None), None), None), None), None), None), None), None), 37), None), None), (...)), None), (...)), None), None), 74), <type at remote 0x8cb440>)) at Objects/object.c:2395
#10 0x00000000005272d8 in Py_DECREF (filename=0x77e260 "./Include/object.h", lineno=602, 
    op=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), 19), None), None), None), None), None), None), None), 24), None), None), None), None), None), None), None), None), 63), None), ()), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), 32), None), None), None), None), None), None), None), None), None), None), None), None), None), None), 37), None), None), (...)), None), (...)), None), None), 74), <type at remote 0x8cb440>)) at ./Include/object.h:527
#11 0x0000000000527326 in Py_XDECREF (
    op=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((0x0, None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), 19), None), None), None), None), None), None), None), 24), None), None), None), None), None), None), None), None), 63), None), ()), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), None), 32), None), None), None), None), None), None), None), None), None), None), None), None), None), None), 37), None), None), (...)), None), (...)), None), None), 74), <type at remote 0x8cb440>)) at ./Include/object.h:602
#12 0x000000000052b369 in insertdict (mp=0x7fa48d0d56d0, key='args', hash=4473388528601007159, value=()) at Objects/dictobject.c:1304
#13 0x000000000052cee8 in _PyDict_SetItem_Take2 (mp=0x7fa48d0d56d0, key='args', value=()) at Objects/dictobject.c:1886
#14 0x000000000052cfa0 in PyDict_SetItem (
    op={'args': (), 'kwargs': {}, 'value': None, '_start_event': <_dummy_event at remote 0x7fa48e5ebb60>, '_notifier': <gevent.libev.corecext.callback at remote 0x7fa48d0c60d0>, '_formatted_info': None, '_links': [], '_ident': None, '_exc_info': (None, None, None), 'spawning_greenlet': None, 'spawn_tree_locals': None, 'spawning_stack': None, '_gevent_local_localimpl_140344724580720': <weakref.ReferenceType at remote 0x7fa48d1317f0>}, key='args', 
    value=()) at Objects/dictobject.c:1906
#15 0x00000000005354c2 in _PyObjectDict_SetItem (tp=0x19c9c40, dictptr=0x7fa48d7756c8, key='args', value=()) at Objects/dictobject.c:5668
#16 0x000000000054890d in _PyObject_GenericSetAttrWithDict (
    obj=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7fa48e5ebb60>, _notifier=<gevent.libev.corecext.callback at remote 0x7fa48d0c60d0>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140344724580720=<weakref.ReferenceType at remote 0x7fa48d1317f0>) at remote 0x7fa48d775630>, name='args', 
    value=(), dict=0x0) at Objects/object.c:1422
#17 0x0000000000548a54 in PyObject_GenericSetAttr (
    obj=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7fa48e5ebb60>, _notifier=<gevent.libev.corecext.callback at remote 0x7fa48d0c60d0>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140344724580720=<weakref.ReferenceType at remote 0x7fa48d1317f0>) at remote 0x7fa48d775630>, name='args', 
    value=()) at Objects/object.c:1456
#18 0x0000000000547ab4 in PyObject_SetAttr (
    v=<Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7fa48e5ebb60>, _notifier=<gevent.libev.corecext.callback at remote 0x7fa48d0c60d0>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140344724580720=<weakref.ReferenceType at remote 0x7fa48d1317f0>) at remote 0x7fa48d775630>, name='args', 
    value=()) at Objects/object.c:1028
#19 0x0000000000610617 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa48cf100a0, throwflag=0) at Python/ceval.c:2889
#20 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa48cf10020, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#21 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48e5f83c0, locals=0x0, args=0x7ffeda5c8ee0, argcount=1, kwnames=0x0) at Python/ceval.c:6428
#22 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48e5f83c0>, stack=0x7ffeda5c8ee0, nargsf=1, kwnames=0x0) at Objects/call.c:393
#23 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48e5f83c0>, args=0x7ffeda5c8ee0, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#24 0x00000000004e09e6 in method_vectorcall (method=<method at remote 0x7fa48d0c38f0>, args=0x9dda90 <_PyRuntime+58928>, nargsf=0, kwnames=0x0) at Objects/classobject.c:67
#25 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fa48d0c38f0>, tuple=(), kwargs=0x0) at Objects/call.c:245
#26 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fa48d0c38f0>, args=(), kwargs=0x0) at Objects/call.c:328
#27 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fa48d0c38f0>, args=(), kwargs=0x0) at Objects/call.c:355
#28 0x00007fa48e616d02 in g_initialstub (mark=0x7ffeda5c9138) at src/greenlet/greenlet.c:960
#29 0x00007fa48e61647e in g_switch (target=0x7fa48d775630, args=(), kwargs=0x0) at src/greenlet/greenlet.c:716
#30 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48e5a6570>, stack=0x7ffeda5c9280, nargsf=2, kwnames=0x0) at Objects/call.c:393
#31 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48e5a6570>, args=0x7ffeda5c9280, nargsf=2, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#32 0x00000000004e0ab1 in method_vectorcall (method=<method at remote 0x7fa48ce38f50>, args=0x7fa48d100248, nargsf=1, kwnames=0x0) at Objects/classobject.c:89
#33 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fa48ce38f50>, tuple=(<Waiter at remote 0x7fa48dc7f830>,), kwargs=0x0)
    at Objects/call.c:245
#34 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fa48ce38f50>, args=(<Waiter at remote 0x7fa48dc7f830>,), kwargs=0x0) at Objects/call.c:328
#35 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fa48ce38f50>, args=(<Waiter at remote 0x7fa48dc7f830>,), kwargs=0x0) at Objects/call.c:355
#36 0x00007fa48e62c55d in gevent_callback (loop=0x7fa48da78f50, callback=<method at remote 0x7fa48ce38f50>, args=(<Waiter at remote 0x7fa48dc7f830>,), watcher=<gevent.libev.corecext.io at remote 0x7fa48d10af90>, 
    c_watcher=0x7fa48d10afd0, revents=1) at src/gevent/libev/callbacks.c:106
#37 0x00007fa48e62c8a1 in gevent_callback_io (_loop=0x7fa48e696da0 <default_loop_struct>, c_watcher=0x7fa48d10afd0, revents=1) at src/gevent/libev/callbacks.c:187
#38 0x00007fa48e63318b in ev_invoke_pending (loop=0x7fa48e696da0 <default_loop_struct>) at /tmp/pip-install-18egfds3/gevent_0a60da4b6e50406b82e7e73be8ea01a3/deps/libev/ev.c:3770
#39 0x00007fa48e633c6d in ev_run (loop=0x7fa48e696da0 <default_loop_struct>, flags=0) at /tmp/pip-install-18egfds3/gevent_0a60da4b6e50406b82e7e73be8ea01a3/deps/libev/ev.c:4190
#40 0x00007fa48e644eb8 in __pyx_pf_6gevent_5libev_8corecext_4loop_14run (__pyx_v_self=0x7fa48da78f50, __pyx_v_nowait=False, __pyx_v_once=False) at src/gevent/libev/corecext.c:9820
#41 0x00007fa48e644d22 in __pyx_pw_6gevent_5libev_8corecext_4loop_15run (__pyx_v_self=<gevent.libev.corecext.loop at remote 0x7fa48da78f50>, __pyx_args=0x7fa48e3e6090, __pyx_nargs=0, __pyx_kwds=0x0)
    at src/gevent/libev/corecext.c:9776
#42 0x00007fa48e678fee in __Pyx_CyFunction_Vectorcall_FASTCALL_KEYWORDS (func=<_cython_3_0_0a11.cython_function_or_method at remote 0x7fa48e7afc50>, args=0x7fa48e3e6090, nargsf=9223372036854775809, kwnames=0x0)
    at src/gevent/libev/corecext.c:27668
#43 0x00000000004dc379 in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<_cython_3_0_0a11.cython_function_or_method at remote 0x7fa48e7afc50>, args=0x7fa48e3e6088, nargsf=9223372036854775809, 
    kwnames=0x0) at ./Include/internal/pycore_call.h:92
#44 0x00000000004dcf80 in PyObject_Vectorcall (callable=<_cython_3_0_0a11.cython_function_or_method at remote 0x7fa48e7afc50>, args=0x7fa48e3e6088, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#45 0x000000000061b9a7 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa48e3e6020, throwflag=0) at Python/ceval.c:4772
#46 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa48e3e6020, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#47 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48e5fadb0, locals=0x0, args=0x7ffeda5cb220, argcount=1, kwnames=0x0) at Python/ceval.c:6428
#48 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48e5fadb0>, stack=0x7ffeda5cb220, nargsf=1, kwnames=0x0) at Objects/call.c:393
#49 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48e5fadb0>, args=0x7ffeda5cb220, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#50 0x00000000004e09e6 in method_vectorcall (method=<method at remote 0x7fa48db94230>, args=0x9dda90 <_PyRuntime+58928>, nargsf=0, kwnames=0x0) at Objects/classobject.c:67
#51 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fa48db94230>, tuple=(), kwargs=0x0) at Objects/call.c:245
#52 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fa48db94230>, args=(), kwargs=0x0) at Objects/call.c:328
#53 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fa48db94230>, args=(), kwargs=0x0) at Objects/call.c:355
#54 0x00007fa48e616d02 in g_initialstub (mark=0x7ffeda5cb478) at src/greenlet/greenlet.c:960
#55 0x00007fa48e61647e in g_switch (target=0x7fa48d68f5b0, args=(), kwargs=0x0) at src/greenlet/greenlet.c:716
#56 0x00007fa48e61798c in green_switch (self=0x7fa48d68f5b0, args=(), kwargs=0x0) at src/greenlet/greenlet.c:1356
#57 0x00000000004eca75 in method_vectorcall_VARARGS_KEYWORDS (func=<method_descriptor at remote 0x7fa48e7f4590>, args=0x7fa49c43cfa8, nargsf=9223372036854775809, kwnames=0x0) at Objects/descrobject.c:364
#58 0x00000000004dc379 in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method_descriptor at remote 0x7fa48e7f4590>, args=0x7fa49c43cfa8, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#59 0x00000000004dcf80 in PyObject_Vectorcall (callable=<method_descriptor at remote 0x7fa48e7f4590>, args=0x7fa49c43cfa8, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#60 0x000000000061b9a7 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43cf40, throwflag=0) at Python/ceval.c:4772
#61 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43cc80, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#62 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48d73f540, locals=0x0, args=0x7fa48d737178, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#63 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48d73f540>, stack=0x7fa48d737178, nargsf=2, kwnames=0x0) at Objects/call.c:393
#64 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4dd24a <_PyFunction_Vectorcall>, callable=<function at remote 0x7fa48d73f540>, tuple=('/tmp/tmpk8vq8qxj', False), kwargs={}) at Objects/call.c:245
#65 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d73f540>, args=('/tmp/tmpk8vq8qxj', False), kwargs={}) at Objects/call.c:328
#66 0x00000000004dd193 in PyObject_Call (callable=<function at remote 0x7fa48d73f540>, args=('/tmp/tmpk8vq8qxj', False), kwargs={}) at Objects/call.c:355
#67 0x00000000006259e7 in do_call_core (tstate=0x9f7e18 <_PyRuntime+166328>, func=<function at remote 0x7fa48d73f540>, callargs=('/tmp/tmpk8vq8qxj', False), kwdict={}, use_tracing=0) at Python/ceval.c:7347
#68 0x000000000061f654 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43cc08, throwflag=0) at Python/ceval.c:5377
#69 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c9a8, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#70 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48da2e570, locals=0x0, args=0x7ffeda5cecf0, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#71 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48da2e570>, stack=0x7ffeda5cecf0, nargsf=2, kwnames=0x0) at Objects/call.c:393
#72 0x00000000004dc8dd in _PyObject_FastCallDictTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48da2e570>, args=0x7ffeda5cecf0, nargsf=2, kwargs=0x0) at Objects/call.c:141
#73 0x00000000004dd66b in _PyObject_Call_Prepend (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48da2e570>, obj=<Popen(returncode=None, finalizer=None, sentinel=None, pid=0) at remote 0x7fa48d7514b0>, args=(<ForkProcess(_identity=(1,), _config={'authkey': <AuthenticationString at remote 0x7fa48e87e150>, 'semprefix': '/mp'}, _parent_pid=509735, _parent_name='MainProcess', _popen=None, _closed=False, _target=<function at remote 0x7fa48d73f540>, _args=('/tmp/tmpk8vq8qxj', False), _kwargs={}, _name='ForkProcess-1') at remote 0x7fa48d7513c0>,), kwargs=0x0) at Objects/call.c:482
#74 0x00000000005733fc in slot_tp_init (self=<Popen(returncode=None, finalizer=None, sentinel=None, pid=0) at remote 0x7fa48d7514b0>, args=(<ForkProcess(_identity=(1,), _config={'authkey': <AuthenticationString at remote 0x7fa48e87e150>, 'semprefix': '/mp'}, _parent_pid=509735, _parent_name='MainProcess', _popen=None, _closed=False, _target=<function at remote 0x7fa48d73f540>, _args=('/tmp/tmpk8vq8qxj', False), _kwargs={}, _name='ForkProcess-1') at remote 0x7fa48d7513c0>,), kwds=0x0) at Objects/typeobject.c:7861
#75 0x0000000000561b5b in type_call (type=0x1d6f440, args=(<ForkProcess(_identity=(1,), _config={'authkey': <AuthenticationString at remote 0x7fa48e87e150>, 'semprefix': '/mp'}, _parent_pid=509735, _parent_name='MainProcess', _popen=None, _closed=False, _target=<function at remote 0x7fa48d73f540>, _args=('/tmp/tmpk8vq8qxj', False), _kwargs={}, _name='ForkProcess-1') at remote 0x7fa48d7513c0>,), kwds=0x0) at Objects/typeobject.c:1112
#76 0x00000000004dcbee in _PyObject_MakeTpCall (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<type at remote 0x1d6f440>, args=0x7fa49c43c9a0, nargs=1, keywords=0x0) at Objects/call.c:214
#77 0x00000000004dc35d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<type at remote 0x1d6f440>, args=0x7fa49c43c9a0, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#78 0x00000000004dcf80 in PyObject_Vectorcall (callable=<type at remote 0x1d6f440>, args=0x7fa49c43c9a0, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#79 0x000000000061b9a7 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c938, throwflag=0) at Python/ceval.c:4772
#80 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c5f0, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#81 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48d949440, locals=0x0, args=0x7ffeda5d09d0, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#82 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48d949440>, stack=0x7ffeda5d09d0, nargsf=2, kwnames=0x0) at Objects/call.c:393
#83 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d949440>, args=0x7ffeda5d09d0, nargsf=2, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#84 0x00000000004e0ab1 in method_vectorcall (method=<method at remote 0x7fa48dc7f9b0>, args=0x7fa48d69eb38, nargsf=1, kwnames=0x0) at Objects/classobject.c:89
#85 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fa48dc7f9b0>, tuple=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs={}) at Objects/call.c:245
#86 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fa48dc7f9b0>, args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs={}) at Objects/call.c:328
#87 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fa48dc7f9b0>, args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs={}) at Objects/call.c:355
#88 0x00000000006259e7 in do_call_core (tstate=0x9f7e18 <_PyRuntime+166328>, func=<method at remote 0x7fa48dc7f9b0>, callargs=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwdict={}, use_tracing=0) at Python/ceval.c:7347
#89 0x000000000061f654 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c568, throwflag=0) at Python/ceval.c:5377
#90 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c568, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#91 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48d949650, locals=0x0, args=0x7ffeda5d26d0, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#92 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48d949650>, stack=0x7ffeda5d26d0, nargsf=2, kwnames=0x0) at Objects/call.c:393
#93 0x00000000004dc8dd in _PyObject_FastCallDictTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d949650>, args=0x7ffeda5d26d0, nargsf=2, kwargs=0x0) at Objects/call.c:141
#94 0x00000000004dd66b in _PyObject_Call_Prepend (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d949650>, obj=<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertListEqual', <type at r...(truncated), args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs=0x0) at Objects/call.c:482
#95 0x0000000000572d3f in slot_tp_call (self=<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertListEqual', <type at r...(truncated), args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwds=0x0) at Objects/typeobject.c:7630
#96 0x00000000004dcbee in _PyObject_MakeTpCall (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertListEqual', <type at r...(truncated), args=0x7fa49c43c550, nargs=1, keywords=0x0) at Objects/call.c:214
#97 0x00000000004dc35d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertListEqual', <type at r...(truncated), args=0x7fa49c43c550, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#98 0x00000000004dcf80 in PyObject_Vectorcall (callable=<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertListEqual', <type at r...(truncated), args=0x7fa49c43c550, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#99 0x000000000061b9a7 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c4c0, throwflag=0) at Python/ceval.c:4772
#100 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c4c0, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#101 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48d94d2e0, locals=0x0, args=0x7ffeda5d4360, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#102 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48d94d2e0>, stack=0x7ffeda5d4360, nargsf=2, kwnames=0x0) at Objects/call.c:393
#103 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d94d2e0>, args=0x7ffeda5d4360, nargsf=2, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#104 0x00000000004e0ab1 in method_vectorcall (method=<method at remote 0x7fa48dd7fd70>, args=0x7fa48d69e9a8, nargsf=1, kwnames=0x0) at Objects/classobject.c:89
#105 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fa48dd7fd70>, tuple=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs={}) at Objects/call.c:245
#106 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fa48dd7fd70>, args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs={}) at Objects/call.c:328
#107 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fa48dd7fd70>, args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs={}) at Objects/call.c:355
#108 0x00000000006259e7 in do_call_core (tstate=0x9f7e18 <_PyRuntime+166328>, func=<method at remote 0x7fa48dd7fd70>, callargs=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwdict={}, use_tracing=0) at Python/ceval.c:7347
#109 0x000000000061f654 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c438, throwflag=0) at Python/ceval.c:5377
#110 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c438, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#111 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48d94d180, locals=0x0, args=0x7ffeda5d6060, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#112 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48d94d180>, stack=0x7ffeda5d6060, nargsf=2, kwnames=0x0) at Objects/call.c:393
#113 0x00000000004dc8dd in _PyObject_FastCallDictTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d94d180>, args=0x7ffeda5d6060, nargsf=2, kwargs=0x0) at Objects/call.c:141
#114 0x00000000004dd66b in _PyObject_Call_Prepend (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d94d180>, obj=<TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertLis...(truncated), args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs=0x0) at Objects/call.c:482
#115 0x0000000000572d3f in slot_tp_call (self=<TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertLis...(truncated), args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwds=0x0) at Objects/typeobject.c:7630
#116 0x00000000004dcbee in _PyObject_MakeTpCall (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertLis...(truncated), args=0x7fa49c43c420, nargs=1, keywords=0x0) at Objects/call.c:214
#117 0x00000000004dc35d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertLis...(truncated), args=0x7fa49c43c420, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#118 0x00000000004dcf80 in PyObject_Vectorcall (callable=<TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at remote 0x8d6500>: 'assertLis...(truncated), args=0x7fa49c43c420, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#119 0x000000000061b9a7 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c390, throwflag=0) at Python/ceval.c:4772
#120 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c390, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#121 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48d94d2e0, locals=0x0, args=0x7ffeda5d7cf0, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#122 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48d94d2e0>, stack=0x7ffeda5d7cf0, nargsf=2, kwnames=0x0) at Objects/call.c:393
#123 0x00000000004e036d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d94d2e0>, args=0x7ffeda5d7cf0, nargsf=2, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#124 0x00000000004e0ab1 in method_vectorcall (method=<method at remote 0x7fa48dbaa7b0>, args=0x7fa48d69e958, nargsf=1, kwnames=0x0) at Objects/classobject.c:89
#125 0x00000000004dcd58 in _PyVectorcall_Call (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x4e088f <method_vectorcall>, callable=<method at remote 0x7fa48dbaa7b0>, tuple=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs={}) at Objects/call.c:245
#126 0x00000000004dd0a4 in _PyObject_Call (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<method at remote 0x7fa48dbaa7b0>, args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs={}) at Objects/call.c:328
#127 0x00000000004dd193 in PyObject_Call (callable=<method at remote 0x7fa48dbaa7b0>, args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs={}) at Objects/call.c:355
#128 0x00000000006259e7 in do_call_core (tstate=0x9f7e18 <_PyRuntime+166328>, func=<method at remote 0x7fa48dbaa7b0>, callargs=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwdict={}, use_tracing=0) at Python/ceval.c:7347
#129 0x000000000061f654 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c308, throwflag=0) at Python/ceval.c:5377
#130 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c308, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#131 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48d94d180, locals=0x0, args=0x7ffeda5d99f0, argcount=2, kwnames=0x0) at Python/ceval.c:6428
#132 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48d94d180>, stack=0x7ffeda5d99f0, nargsf=2, kwnames=0x0) at Objects/call.c:393
#133 0x00000000004dc8dd in _PyObject_FastCallDictTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d94d180>, args=0x7ffeda5d99f0, nargsf=2, kwargs=0x0) at Objects/call.c:141
#134 0x00000000004dd66b in _PyObject_Call_Prepend (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d94d180>, obj=<TestSuite(_tests=[None, <TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at rem...(truncated), args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwargs=0x0) at Objects/call.c:482
#135 0x0000000000572d3f in slot_tp_call (self=<TestSuite(_tests=[None, <TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at rem...(truncated), args=(<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>,), kwds=0x0) at Objects/typeobject.c:7630
#136 0x00000000004dcbee in _PyObject_MakeTpCall (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSuite(_tests=[None, <TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at rem...(truncated), args=0x7fa49c43c2e0, nargs=1, keywords=0x0) at Objects/call.c:214
#137 0x00000000004dc35d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<TestSuite(_tests=[None, <TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at rem...(truncated), args=0x7fa49c43c2e0, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#138 0x00000000004dcf80 in PyObject_Vectorcall (callable=<TestSuite(_tests=[None, <TestSuite(_tests=[<TestSyncManager(_testMethodName='test_manager_fork', _outcome=<_Outcome(expecting_failure=False, result=<TextTestResult(failfast=False, failures=[], errors=[], testsRun=1, skipped=[], expectedFailures=[], unexpectedSuccesses=[], shouldStop=False, buffer=False, tb_locals=False, _stdout_buffer=None, _stderr_buffer=None, _original_stdout=<_io.TextIOWrapper at remote 0x7fa48e906210>, _original_stderr=<_io.TextIOWrapper at remote 0x7fa48e906300>, _mirrorOutput=False, stream=<_WritelnDecorator(stream=<_io.TextIOWrapper at remote 0x7fa48e906300>) at remote 0x7fa48d750ab0>, showAll=False, dots=True, descriptions=True, _newline=True, _testRunEntered=True, _moduleSetUpFailed=False, _previousTestClass=<type at remote 0x1e700f0>) at remote 0x7fa48e602040>, result_supports_subtests=True, success=True, expectedFailure=None) at remote 0x7fa48d9a2b80>, _testMethodDoc=None, _cleanups=[], _subtest=None, _type_equality_funcs={<type at remote 0x8d76c0>: 'assertDictEqual', <type at rem...(truncated), args=0x7fa49c43c2e0, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:299
#139 0x000000000061b9a7 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c200, throwflag=0) at Python/ceval.c:4772
#140 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c090, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#141 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48d6ad230, locals=0x0, args=0x7ffeda5db6a0, argcount=1, kwnames=0x0) at Python/ceval.c:6428
#142 0x00000000004dd334 in _PyFunction_Vectorcall (func=<function at remote 0x7fa48d6ad230>, stack=0x7ffeda5db6a0, nargsf=1, kwnames=0x0) at Objects/call.c:393
#143 0x00000000004dc8dd in _PyObject_FastCallDictTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d6ad230>, args=0x7ffeda5db6a0, nargsf=1, kwargs=0x0) at Objects/call.c:141
#144 0x00000000004dd66b in _PyObject_Call_Prepend (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<function at remote 0x7fa48d6ad230>, obj=<TestProgram(module=<module at remote 0x7fa48e933e90>, exit=True, failfast=False, catchbreak=False, verbosity=1, buffer=False, tb_locals=False, warnings='default', defaultTest=None, testRunner=<type at remote 0x1e29d20>, testLoader=<TestLoader(errors=[], _loading_packages=set()) at remote 0x7fa48d946a40>, progName='monkey_manager_tests.py', testNamePatterns=[], _main_parser=<ArgumentParser(description=None, argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x1e1f4b0>, 'store': <type at remote 0x1e1f4b0>, 'store_const': <type at remote 0x1e1fbf0>, 'store_true': <type at remote 0x1e20330>, 'store_false': <type at remote 0x1e20a70>, 'append': <type at remote 0x1e211b0>, 'append_const': <type at remote 0x1e218f0>, 'count': <type at remote 0x1e22030>, 'help': <type at remote 0x1e22770>, 'version': <type at remote 0x1e234e0>, 'parsers': <type at remote 0x1e24020>, 'extend': <type at remote 0x1e24760>}, 'type': {None: <function at remote 0x7fa48d73fac0>}...(truncated), args=(), kwargs=0x0) at Objects/call.c:482
#145 0x00000000005733fc in slot_tp_init (self=<TestProgram(module=<module at remote 0x7fa48e933e90>, exit=True, failfast=False, catchbreak=False, verbosity=1, buffer=False, tb_locals=False, warnings='default', defaultTest=None, testRunner=<type at remote 0x1e29d20>, testLoader=<TestLoader(errors=[], _loading_packages=set()) at remote 0x7fa48d946a40>, progName='monkey_manager_tests.py', testNamePatterns=[], _main_parser=<ArgumentParser(description=None, argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x1e1f4b0>, 'store': <type at remote 0x1e1f4b0>, 'store_const': <type at remote 0x1e1fbf0>, 'store_true': <type at remote 0x1e20330>, 'store_false': <type at remote 0x1e20a70>, 'append': <type at remote 0x1e211b0>, 'append_const': <type at remote 0x1e218f0>, 'count': <type at remote 0x1e22030>, 'help': <type at remote 0x1e22770>, 'version': <type at remote 0x1e234e0>, 'parsers': <type at remote 0x1e24020>, 'extend': <type at remote 0x1e24760>}, 'type': {None: <function at remote 0x7fa48d73fac0>}...(truncated), args=(), kwds=0x0) at Objects/typeobject.c:7861
#146 0x0000000000561b5b in type_call (type=0x1e2aae0, args=(), kwds=0x0) at Objects/typeobject.c:1112
#147 0x00000000004dcbee in _PyObject_MakeTpCall (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<type at remote 0x1e2aae0>, args=0x7fa49c43c078, nargs=0, keywords=0x0) at Objects/call.c:214
#148 0x00000000004dc35d in _PyObject_VectorcallTstate (tstate=0x9f7e18 <_PyRuntime+166328>, callable=<type at remote 0x1e2aae0>, args=0x7fa49c43c078, nargsf=9223372036854775808, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#149 0x00000000004dcf80 in PyObject_Vectorcall (callable=<type at remote 0x1e2aae0>, args=0x7fa49c43c078, nargsf=9223372036854775808, kwnames=0x0) at Objects/call.c:299
#150 0x000000000061b9a7 in _PyEval_EvalFrameDefault (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c020, throwflag=0) at Python/ceval.c:4772
#151 0x0000000000606070 in _PyEval_EvalFrame (tstate=0x9f7e18 <_PyRuntime+166328>, frame=0x7fa49c43c020, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#152 0x00000000006234a4 in _PyEval_Vector (tstate=0x9f7e18 <_PyRuntime+166328>, func=0x7fa48e9b0c00, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fa48e935c30>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa49c2fb6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fa48e86b750>, 'monkey': <module at remote 0x7fa48e844b30>, 'TestCase': <type at remote 0x1dfa340>, 'main': <type at remote 0x1e2aae0>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fa48d8ff7d0>, 'spawn': <method at remote 0x7fa48e7c65d0>, 'wait': <function at remote 0x7fa48e5e0260>, 'get_hub': <function at remote 0x7fa48e5a5390>, 'sleep': <function at remote 0x7fa48e5f9440>, 'mp': <module at remote 0x7fa48e844bf0>, 'mktemp': <function at remote 0x...(truncated), args=0x0, argcount=0, kwnames=0x0) at Python/ceval.c:6428
#153 0x0000000000609163 in PyEval_EvalCode (co=<code at remote 0x18dcbe0>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fa48e935c30>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa49c2fb6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fa48e86b750>, 'monkey': <module at remote 0x7fa48e844b30>, 'TestCase': <type at remote 0x1dfa340>, 'main': <type at remote 0x1e2aae0>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fa48d8ff7d0>, 'spawn': <method at remote 0x7fa48e7c65d0>, 'wait': <function at remote 0x7fa48e5e0260>, 'get_hub': <function at remote 0x7fa48e5a5390>, 'sleep': <function at remote 0x7fa48e5f9440>, 'mp': <module at remote 0x7fa48e844bf0>, 'mktemp': <function at remote 0x...(truncated), locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fa48e935c30>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa49c2fb6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fa48e86b750>, 'monkey': <module at remote 0x7fa48e844b30>, 'TestCase': <type at remote 0x1dfa340>, 'main': <type at remote 0x1e2aae0>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fa48d8ff7d0>, 'spawn': <method at remote 0x7fa48e7c65d0>, 'wait': <function at remote 0x7fa48e5e0260>, 'get_hub': <function at remote 0x7fa48e5a5390>, 'sleep': <function at remote 0x7fa48e5f9440>, 'mp': <module at remote 0x7fa48e844bf0>, 'mktemp': <function at remote 0x...(truncated)) at Python/ceval.c:1154
#154 0x0000000000686eba in run_eval_code_obj (tstate=0x9f7e18 <_PyRuntime+166328>, co=0x18dcbe0, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fa48e935c30>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa49c2fb6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fa48e86b750>, 'monkey': <module at remote 0x7fa48e844b30>, 'TestCase': <type at remote 0x1dfa340>, 'main': <type at remote 0x1e2aae0>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fa48d8ff7d0>, 'spawn': <method at remote 0x7fa48e7c65d0>, 'wait': <function at remote 0x7fa48e5e0260>, 'get_hub': <function at remote 0x7fa48e5a5390>, 'sleep': <function at remote 0x7fa48e5f9440>, 'mp': <module at remote 0x7fa48e844bf0>, 'mktemp': <function at remote 0x...(truncated), locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fa48e935c30>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa49c2fb6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fa48e86b750>, 'monkey': <module at remote 0x7fa48e844b30>, 'TestCase': <type at remote 0x1dfa340>, 'main': <type at remote 0x1e2aae0>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fa48d8ff7d0>, 'spawn': <method at remote 0x7fa48e7c65d0>, 'wait': <function at remote 0x7fa48e5e0260>, 'get_hub': <function at remote 0x7fa48e5a5390>, 'sleep': <function at remote 0x7fa48e5f9440>, 'mp': <module at remote 0x7fa48e844bf0>, 'mktemp': <function at remote 0x...(truncated)) at Python/pythonrun.c:1714
#155 0x0000000000686fa6 in run_mod (mod=0x18f12f0, filename='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fa48e935c30>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa49c2fb6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fa48e86b750>, 'monkey': <module at remote 0x7fa48e844b30>, 'TestCase': <type at remote 0x1dfa340>, 'main': <type at remote 0x1e2aae0>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fa48d8ff7d0>, 'spawn': <method at remote 0x7fa48e7c65d0>, 'wait': <function at remote 0x7fa48e5e0260>, 'get_hub': <function at remote 0x7fa48e5a5390>, 'sleep': <function at remote 0x7fa48e5f9440>, 'mp': <module at remote 0x7fa48e844bf0>, 'mktemp': <function at remote 0x...(truncated), locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fa48e935c30>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa49c2fb6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fa48e86b750>, 'monkey': <module at remote 0x7fa48e844b30>, 'TestCase': <type at remote 0x1dfa340>, 'main': <type at remote 0x1e2aae0>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fa48d8ff7d0>, 'spawn': <method at remote 0x7fa48e7c65d0>, 'wait': <function at remote 0x7fa48e5e0260>, 'get_hub': <function at remote 0x7fa48e5a5390>, 'sleep': <function at remote 0x7fa48e5f9440>, 'mp': <module at remote 0x7fa48e844bf0>, 'mktemp': <function at remote 0x...(truncated), flags=0x7ffeda5dd528, arena=0x7fa48e827220) at Python/pythonrun.c:1735
#156 0x0000000000686c96 in pyrun_file (fp=0x18c5140, filename='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', start=257, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fa48e935c30>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa49c2fb6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fa48e86b750>, 'monkey': <module at remote 0x7fa48e844b30>, 'TestCase': <type at remote 0x1dfa340>, 'main': <type at remote 0x1e2aae0>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fa48d8ff7d0>, 'spawn': <method at remote 0x7fa48e7c65d0>, 'wait': <function at remote 0x7fa48e5e0260>, 'get_hub': <function at remote 0x7fa48e5a5390>, 'sleep': <function at remote 0x7fa48e5f9440>, 'mp': <module at remote 0x7fa48e844bf0>, 'mktemp': <function at remote 0x...(truncated), locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py') at remote 0x7fa48e935c30>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fa49c2fb6b0>, '__file__': '/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', '__cached__': None, 'current_process': <function at remote 0x7fa48e86b750>, 'monkey': <module at remote 0x7fa48e844b30>, 'TestCase': <type at remote 0x1dfa340>, 'main': <type at remote 0x1e2aae0>, 'GEVENT_SAVED_MODULE_SETTINGS': '_gevent_saved_patch_all_module_settings', 'trace': <module at remote 0x7fa48d8ff7d0>, 'spawn': <method at remote 0x7fa48e7c65d0>, 'wait': <function at remote 0x7fa48e5e0260>, 'get_hub': <function at remote 0x7fa48e5a5390>, 'sleep': <function at remote 0x7fa48e5f9440>, 'mp': <module at remote 0x7fa48e844bf0>, 'mktemp': <function at remote 0x...(truncated), closeit=1, flags=0x7ffeda5dd528) at Python/pythonrun.c:1630
#157 0x0000000000684553 in _PyRun_SimpleFileObject (fp=0x18c5140, filename='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', closeit=1, flags=0x7ffeda5dd528) at Python/pythonrun.c:440
#158 0x0000000000683ab0 in _PyRun_AnyFileObject (fp=0x18c5140, filename='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', closeit=1, flags=0x7ffeda5dd528) at Python/pythonrun.c:79
#159 0x00000000006afeaf in pymain_run_file_obj (program_name='/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/bin/python3.11', filename='/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py', skip_source_first_line=0) at Modules/main.c:360
#160 0x00000000006aff66 in pymain_run_file (config=0x9dde60 <_PyRuntime+59904>) at Modules/main.c:379
#161 0x00000000006b0595 in pymain_run_python (exitcode=0x7ffeda5dd6ac) at Modules/main.c:601
#162 0x00000000006b0698 in Py_RunMain () at Modules/main.c:680
#163 0x00000000006b072c in pymain_main (args=0x7ffeda5dd710) at Modules/main.c:710
#164 0x00000000006b07a6 in Py_BytesMain (argc=2, argv=0x7ffeda5dd868) at Modules/main.c:734
#165 0x000000000041d7e6 in main (argc=2, argv=0x7ffeda5dd868) at ./Programs/python.c:15
(gdb) py-bt
Traceback (most recent call first):
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/greenlet.py", line 927, in __free
    self.args = ()
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/greenlet.py", line 914, in run
    self.__free()
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/hub.py", line 647, in run
    loop.run()
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/_greenlet_primitives.py", line 65, in switch
    return _greenlet_switch(self) # pylint:disable=undefined-variable
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/_waiter.py", line 154, in get
    return self.hub.switch()
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/hub.py", line 742, in join
    waiter.get()
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/gevent/_hub_primitives.py", line 286, in wait_on_objects
    return hub.join(timeout=timeout) # pylint:disable=
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/_mp_manager_server.py", line 130, in _manager_process
    wait(timeout=300)
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/_mp_manager_server.py", line 90, in manager_process
    _manager_process(addr)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/popen_fork.py", line 71, in _launch
    code = process_obj._bootstrap(parent_sentinel=child_r)
  File "/home/arcivanov/Documents/src/karellen/geventmp/target/venv/test/cpython-3.11.0.candidate.2/lib/python3.11/site-packages/geventmp/_mp/3/_mp_popen_fork.py", line 27, in _launch
    super()._launch(process_obj)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/context.py", line 281, in _Popen
    return Popen(process_obj)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 85, in _test_manager
    p.start()
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 81, in run_manager_test
    self._test_manager(ctx, remote_trace)
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 71, in test_manager_fork
    self.run_manager_test("fork")
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/case.py", line 623, in run
    self._callTestMethod(testMethod)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/case.py", line 678, in __call__
    return self.run(*args, **kwds)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/suite.py", line 122, in run
    test(result)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/suite.py", line 122, in run
    test(result)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/runner.py", line 217, in run
    test(result)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/main.py", line 274, in runTests
    self.result = testRunner.run(self.test)
  File "/home/arcivanov/.pyenv/versions/3.11.0rc2-debug/lib/python3.11/unittest/main.py", line 102, in __init__
    self.runTests()
  File "/home/arcivanov/Documents/src/karellen/geventmp/src/integrationtest/python/monkey_manager_tests.py", line 130, in <module>
    main()
(gdb) py-locals
Locals for __free
self = <Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7fa48e5ebb60>, _notifier=<gevent.libev.corecext.callback at remote 0x7fa48d0c60d0>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140344724580720=<weakref.ReferenceType at remote 0x7fa48d1317f0>) at remote 0x7fa48d775630>
Locals for run
self = <Greenlet(args=(), kwargs={}, value=None, _start_event=<_dummy_event at remote 0x7fa48e5ebb60>, _notifier=<gevent.libev.corecext.callback at remote 0x7fa48d0c60d0>, _formatted_info=None, _links=[], _ident=None, _exc_info=(None, None, None), spawning_greenlet=None, spawn_tree_locals=None, spawning_stack=None, _gevent_local_localimpl_140344724580720=<weakref.ReferenceType at remote 0x7fa48d1317f0>) at remote 0x7fa48d775630>
result = None

Oct 12 '22 21:10 arcivanov

Full dump of all objects in trashcan:

gc.txt

Oct 12 '22 21:10 arcivanov

The reason frame got excluded from having their reference count temporarily bumped to be dumped is because there occurs an assertion in code object here if they are not:

https://github.com/python/cpython/blob/450306ed6770d7ffd16cb9128e471677dda9b3d1/Objects/codeobject.c#L771

Oct 12 '22 21:10 arcivanov

@jamadden does greenlet by any chance do anything funky with traceback?

Oct 12 '22 22:10 arcivanov