Feature Request: Get failed value (instead of outputting to IO)

[flip111]

trafficstars

I saw this SO post https://stackoverflow.com/questions/8191131/find-the-value-that-failed-for-quickcheck

is it possible to have a function like this build into quickcheck itself?

My use case is that i have a parser and a pretty printer and the test flow is like this:

quickcheck generate random syntax tree -> print -> source code -> parse -> syntax tree (should be same as input)

If a test case fails i get the syntax tree which is hard to read and i like to try to send it to the printer.

[nick8325]

Hi,

The trouble is, what type should the counterexample have?

quickCheckWithCounterexample :: Property -> ???

For this feature to work, the type of properties would need to be enriched to encode which types were quantified over. This change would of course be extremely backwards-incompatible. For now, the best thing is to make your property write to an IORef and read it back afterwards.

For simple properties which don't use any of the property combinators like forAll, I have some code which automates the IORef trick, by defining a WithCounterexample class parallel to the existing Testable class:

{-# LANGUAGE TypeFamilies #-}
module Counterexample where

import Test.QuickCheck
import Data.IORef

class Testable a => WithCounterexample a where
  type Counterexample a
  withCounterexample :: (Counterexample a -> IO ()) -> a -> Property

instance WithCounterexample Bool where
  type Counterexample Bool = ()
  withCounterexample f prop = whenFail (f ()) prop

instance WithCounterexample Property where
  type Counterexample Property = ()
  withCounterexample f prop = whenFail (f ()) prop

instance (Show a, Arbitrary a, WithCounterexample b) => WithCounterexample (a -> b) where
  type Counterexample (a -> b) = (a, Counterexample b)
  withCounterexample f prop =
    forAllShrink arbitrary shrink $ \x ->
      withCounterexample (\y -> f (x, y)) (prop x)

quickCheckCounterexample :: WithCounterexample prop => prop -> IO (Maybe (Counterexample prop))
quickCheckCounterexample prop = do
  ref <- newIORef (error "counterexample not available")
  res <- quickCheckResult (withCounterexample (writeIORef ref) prop)
  case res of
    Failure{} ->
      fmap Just (readIORef ref)
    _ -> return Nothing

You can use it like this:

-- Example property.
prop_reverse :: [Int] -> [Int] -> Bool
prop_reverse xs ys = reverse (xs ++ ys) == reverse xs ++ reverse ys

-- Run QuickCheck and get the counterexample out.
reverseCounterexample :: IO (Maybe ([Int], ([Int], ())))
reverseCounterexample = quickCheckCounterexample prop_reverse

Perhaps this code should go into QuickCheck somewhere but then again it's quite limited.

[nick8325]

I polished this idea a bit and put it into a library, which is now on Hackage:

http://hackage.haskell.org/package/quickcheck-with-counterexamples

[flip111]

Thank you @nick8325 for this great library! Would it be possible to add it to stackage? I would like to try the library with stack.

[sjakobi]

@flip111:

Would it be possible to add it to stackage? I would like to try the library with stack.

In case you aren't aware, you can easily use a non-snapshot package with stack by adding it to the extra-deps section of your stack.yaml.

In this case this would look like:

<resolver, packages etc>
extra-deps:
- quickcheck-with-counterexamples-1.0

The stack.yaml for your global / default project is typically at ~/.stack/global-project/stack.yaml.

[nick8325]

Would it be possible to add it to stackage? I would like to try the library with stack.

I wouldn't mind doing so in the future, but I would like to wait for a bit in case there are any serious bugs or whatever.

[Lysxia]

How about providing counterexamples as Dynamic values? Just throwing the idea out there for the moment.

[amigalemming]

I used sample' for poor man's counterexample finding: #310.

[MaximilianAlgehed]

We have a new feature in the pipeline for addressing this in #376