Add soft timeouts

[noschinl]

Implements #86 by adding a soft-timeout option. In contrast to the existing timeout option, a soft timeout is not asynchronous and therefore allows the test runner to intercept the timeout exception.

[noschinl]

I ran a few tests in with QC and this seemed to work fine.

[UnkindPartition]

I'm somewhat hesitant to have two different timeout mechanisms. Do you see a way to have a single option to do the right thing in all cases? (I don't have time to dive into the code at the moment.)

[noschinl]

I'll think about it. Can you remember what the reason for 46db7724 was?

In particular: Why is the timeout applied to the wait, not to the action?

[UnkindPartition]

This is because some providers catch and ignore the exception. I don't remember which one in particular was the culprit — maybe the old HUnit, maybe quickcheck.

So the ideal solution could be to apply two timeouts: one to the testing action itself, and the other (say, 1 second later) to the thread, just in case the former was ignored.

Also, if we apply the "soft" timeout, as you call it, then it will be no longer recognized and reported as a timeout, it'll be just quickcheck's failure. Which is fine, I guess, but we probably should apply this soft timeout to quickcheck only, so that other timeouts are recognized as such.

Furthermore, it means the the thread timeout should be T+1 for quickcheck and T for other providers.

Now this got pretty complicated... Any hope for a simpler design?

[noschinl]

I encountered some problems with this patch. In some cases, even the classic hard timeouts stop working when soft timeouts are enabled. I haven't had time to investigate this yet, but as it stands, it seems to be broken.

Am 23. Oktober 2014 21:29:08 MESZ, schrieb Roman Cheplyaka [email protected]:

I'm somewhat hesitant to have to different timeout mechanisms. Do you see a way to have a single option to do the right thing in all cases? (I don't have time to dive into the code at the moment.)

---

Reply to this email directly or view it on GitHub: https://github.com/feuerbach/tasty/pull/89#issuecomment-60295187

[noschinl]

Had some time to look a bit closer: It's not the hard timeout failing, but the rendering the result returned by QuickCheck does not terminate.

That is because we use some combinator which makes the result depending on the non-terminating computation:

import Control.DeepSeq (NFData)
import Control.Spoon (spoon)
import Test.QuickCheck (Property)
import Test.QuickCheck.Property (counterexample)

infix 4 ==?

(==?) :: (Show b, Eq b, NFData b) => b -> b -> Property
(==?) user ref = counterexample msg $ user == ref
  where msg =
          "### Expected output:\n" ++ show ref ++ "\n" ++
          case spoon user of
            Just res -> "### Actual output:\n" ++ show res ++ "\n"
            Nothing -> "### Exception thrown"

I wonder whether a provision for this case would belong to tasty or to tasty-quickcheck -- I don't know whether there are any other testing tools which could reasonably return a non-terminating result?

[UnkindPartition]

We have https://github.com/feuerbach/tasty/blob/master/core/Test/Tasty/Runners/Utils.hs#L16, could your thing go in that function?

[noschinl]

Oh, totally forgot about this function. Yes, I could put this in there, if you are fine with adding a short (say 1s or 0.5s) timeout.

Roman Cheplyaka [email protected] schrieb:

We have https://github.com/feuerbach/tasty/blob/master/core/Test/Tasty/Runners/Utils.hs#L16, could your thing go in that function?

— Reply to this email directly or view it on GitHub.

[UnkindPartition]

Yes, I think it makes sense.

[UnkindPartition]

@noschinl could you remind me on the status of this PR? I vaguely remember that it wasn't ready for some reason — is that right? Has anything changed?

[noschinl]

Thanks for reminding me. As mentioned above, soft timeouts need timeouts in formatMessage. I have updated the pull request accordingly. We have been using both soft-timeouts and hard-timeouts simultaneously for about three months and timeouts worked properly.

There is one open issue, namely that in combination with soft timeouts, Tasty does not necessarily recognize a timeout as a timeout, but as a failed test. I'd need to look into the details again, but the fix hinges on basvandijk/unbounded-delays#5, on which I'll need to follow up.

[Bodigrim]

A test provider can look at the hard timeout, derive from it a soft timeout (like, 10% less) and apply it internally. Thus I don't see a good justification to complicate things globally. Unless there are specific objections voiced within two weeks, I intend to close this PR.