pytest-trio icon indicating copy to clipboard operation
pytest-trio copied to clipboard

Add test timeout support

Open njsmith opened this issue 7 years ago • 2 comments

This will need some help from Trio – see https://github.com/python-trio/trio/issues/168

njsmith avatar Jul 24 '18 09:07 njsmith

Actually.... this doesn't need help from Trio, if we're clever. It's not the most elegant thing, but we can start a timer thread running, and then if the timeout expires it can call back into Trio. Of course we'll also want to be able to kill it early, so something like, a thread blocked in select would work.

def timeout_thread(wake_sock, timeout, trio_token, cancel_scope):
    readable, _, _ = select.select([wake_sock], [], [], timeout)
    if not readable:
        # we were woken by the timeout expiring
        try:
            trio_token.run_sync_soon(cancel_scope.cancel)
        except trio.RunFinishedError:
            pass

async with trio.open_nursery() as nursery:
    a, b = socket.socketpair()  # blocking sockets
    trio_token = trio.hazmat.current_trio_token()
    try:
        with trio.open_cancel_scope() as timeout_cancel_scope:
            nursery.start_soon(trio.run_sync_in_worker_thread, timeout_thread, b, timeout, trio_token, timeout_cancel_scope, limiter=UNLIMITED)
            # ... do actual test here ...
    finally:
        a.send(b"x")
        if timeout_cancel_scope.cancelled_caught:
            raise ...

This does require that the Trio scheduler be functioning properly, but so would a more intrusive version baked into the Trio scheduler. This also has the advantage that by being in a third-party library we have a lot of flexibility to adjust the response to the timeout however we want – e.g. instead of just cancelling the test, we could have it capture and print a snapshot of the task tree with stacktraces. Or try cancelling after X seconds, and then if the test is still running after another Y seconds (i.e. it's ignoring the cancellation), use harsher methods to disable pytest's output capturing, dump debugging information to the screen, and then call os._exit() to crash the process.

njsmith avatar Jul 26 '18 02:07 njsmith

In python-trio/trio#168 I suggested that we might want to have two separate timeouts – an idle timeout and a global timeout:

What twisted uses is a simple global timeout for the whole test, after which I assume it just stops executing, abandoning any callback chains mid-flight. For many purposes an idle-time timeout makes more sense ("if all tasks have been asleep for X seconds, cancel the test") - in particular, one can set a pretty aggressive timeout for this so that frozen tests fail fast, but big complicated tests that just do a lot of stuff won't trigger it. However, we probably also want a single cumulative timeout ("if the test started X seconds ago, cancel it"), to catch cases like the ssl test that's mentioned in bpo-30437, where a misbehaving test was constantly sending data in an infinite loop, so it never went idle but would have responded to an absolute timeout.

On further consideration, I think a deadlock detector like in https://github.com/python-trio/trio/issues/1085 would be a better solution for the "idle timeout" use cases, so maybe pytest-trio should just focus on providing a global timeout.

njsmith avatar Jun 07 '19 04:06 njsmith