Need way to perform work upon test failure

[newhoggy]

In particular, when a test fails, I'd like a way to perform some IO action to read a file and annotate it.

It is insufficient to do the annotation before the assertion because the file is a log file that is growing and will not contain all the information I need, hence the desire to add the annotation at the point of failure.

[ocharles]

It sounds like all you want is onException from the exceptions library. I believe test failure is just raising an exception

On Wed, 16 Sep 2020, at 12:07 AM, John Ky wrote:

In particular, when a test fails, I'd like a way to perform some IO action to read a file and annotate it.

It is insufficient to do the annotation before the assertion because the file is a log file that is growing and will not contain all the information I need, hence the desire to add the annotation at the point of failure.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hedgehogqa/haskell-hedgehog/issues/399, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAFDDWFAZPKYVB3OC7LKRLSF7XTBANCNFSM4RN4ZCDA.

[newhoggy]

Are exceptions considered part of the API or an implementation detail?

[HuwCampbell]

newtype PropertyT m a =
  PropertyT {
      unPropertyT :: TestT (GenT m) a
    }

newtype TestT m a =
  TestT {
      unTest :: ExceptT Failure (Lazy.WriterT Journal m) a
    }

It doesn't use exceptions for failures, though it does catch them. But you can unwrap and rewrap these types as much as you like.

[newhoggy]

I got it working:

               ┏━━ test/Spec/Network.hs ━━━
            76 ┃ hprop_foo :: Property
            77 ┃ hprop_foo = H.propertyOnce . H.runFinallies $ do
            78 ┃   H.onFailure $ H.noteM_ $ H.readFile "moo.txt"
               ┃   │ Reading file: moo.txt
               ┃   │ hello world
            79 ┃ 
            80 ┃   H.writeFile "moo.txt" "hello world"
               ┃   │ Writing file: moo.txt
            81 ┃ 
            82 ┃   H.failure
               ┃   ^^^^^^^^^

Line 78 says if the test fails, run provided action, which is annotate the contents of a file. Line 80 writes the file. Line 82 fails, causing the provided action to run. This only works because the writeFile has already happened.

[moodmosaic]

@newhoggy, great :+1: Feel free to paste an example here, if that's easy for you, for anyone else that might be looking into this in the future.

[newhoggy]

This is the code I used to expose the ability to catch and throw assertions:

https://github.com/input-output-hk/cardano-node/blob/master/hedgehog-extras/src/Hedgehog/Extras/Test/MonadAssertion.hs

The runFinallies function is defined here:

https://github.com/input-output-hk/cardano-node/blob/5106f637233a9f2c12faaca571b78378714b4d0a/hedgehog-extras/src/Hedgehog/Extras/Test/Base.hs#L362

And onFailure function is defined here:

https://github.com/input-output-hk/cardano-node/blob/5106f637233a9f2c12faaca571b78378714b4d0a/hedgehog-extras/src/Hedgehog/Extras/Test/Base.hs#L349

[newhoggy]

I'd love for something like this to be upstreamed into hedgehog.