Request for tutorial: How to use MonadGen, GenT, TestT, and PropertyT in custom monad stack

[saurabhnanda]

trafficstars

Context

My question is similar to https://github.com/hedgehogqa/haskell-hedgehog/issues/246 and I've probably already touched upon a related topic in https://github.com/hedgehogqa/haskell-hedgehog/issues/311

This is the second time I am attempting to use property-based testing in my codebase, and I seem to be struggling again. The first time was 3 years ago with QuickCheck

Here's the kind of property that I am, broadly, trying to write:

• Open a DB connection, prepare my AppM's environment, and start a DB txn.
• Generate a random in-memory ValueA
• Save ValueA to the DB (ideally I'd like to use a function in my AppM monad, instead of IO)
• Generate a random in-memory ChildOfValueA that depends on ValueA
• Save ChildOfValueA to the DB
• Invoke my domain method (again, in AppM) with (ValueA, ChildOfValueA) to get DomainResult
• Assert something on DomainResult
• Fetch some CriticalDBState from the DB (again, in AppM)
• Assert something on CriticalDBState
• Rollback the DB txn from the very first step.

I've tried messing around with property, test, hoist, resourcet, etc and seem to hitting some impedance mismatch, or have a misunderstanding of Hedgehog's structure/internals.

I noticed an interesting development between hedgehog-0.6 [1] and hedgehog-1.0 wrt MonadGen, which considered alongwith, TestT, PropertyT, and GenT, seems to indicate that there is some way in which the following "effects" or "abilities" can be "injected" into my app's AppM monad:

• Generating/shrinking random values
• Asserting expectations

Questions

1. What is the correct way to use MonadGen, GenT, TestT, and PropertyT in one's own monad-transformer stack?

2. Related: Is there any way that properties can be written in the following manner:

-- NOTE: This is hypothetical code

myProp appEnv = runAppM appEnv $ property $ do     -- or `property $ runAppM appEnv $ do`
  valueA <- DB.save =<< (forAll MyGen.valueA)
  childOfValueA <- DB.save =<< (forAll $ MyGen.childOfValueA valueA)
  domainResult <- domainFunction (valueA, childOfValueA)
  assert (isJust domainResult)
  criticalDbState <- DB.fetch something
  assert (isJust criticalDbState)

3. Possibly related: Why is there no MonadMask instance for GenT m?

[1] The version in Stack LTS-12.1, which my codebase uses

[jacobstanley]

Thanks for taking the time to write this up, let's see if we can get you going with some back and forth.

1. I'm not entirely sure what you're meaning by this, but maybe we can dig in to it. What kind of Monad is AppM for a start, just ReaderT Env?

2. I would take a guess that property . hoist somethingToCreateAndRunAppM $ do might work for you. If you have a small example somewhere project somewhere I'd be happy to try and get it working through it with you.

3. GenT m is a tree monad transformer so any control flow like operations are very difficult (impossible?) to implement. It's for the same reason that pipes cannot implement MonadMask. I'd love for someone to prove me wrong, I've burned countless days trying to solve that. Also MonadBaseControl / MonadTransControl.

Fwiw, I don't think you really will need anything other than Gen (i.e. GenT Identity) to do what you're trying to do. I'd be hopefully we can solve all your open issues together.

I would recommend not doing any of the DB IO until after the forAll block if at all possible. It can be workable but the way IO works during shrinking can be quite confusing.

Is there a reason why you can't do the forAlls first, and all the IO after (including runAppM)?

[saurabhnanda]

Thanks @jacobstanley for offering to help. Much appreciated 😄

I'm not entirely sure what you're meaning by this, but maybe we can dig in to it. What kind of Monad is AppM for a start, just ReaderT Env?

Here's the actual definition of AppM from my codebase:

newtype AppM a = AppM (ReaderT Env IO a)
  deriving (Functor, Applicative, Monad, MonadReader Env, MonadIO, MonadThrow, MonadCatch, MonadMask, MonadUnliftIO)

I would take a guess that property . hoist somethingToCreateAndRunAppM $ do might work for you. If you have a small example somewhere project somewhere I'd be happy to try and get it working through it with you.

Give me some time to put together a repo with minimal dependencies, but with the essential structure of the actual code.

I would recommend not doing any of the DB IO until after the forAll block if at all possible. It can be workable but the way IO works during shrinking can be quite confusing.

This is because, apart from other randomly generated values, childOfValueA contains the primary-key of valueA, which can be known only once valueA is saved to the DB. I tried getting around this, by a clever hack...

randomValueA :: (MonadGen m) => m UnsavedValueA

randomChildOfValueA :: (MonadGen m) => m (SavedValueA -> UnsavedChildOfValueA)

... but it seems that hedgehog requires a Show instance on all randomly generated values (and therefore it can't generate random functions). A Google search led me to hedgehog-fn but I was intimidated by the type-hackery involved, and figured that I might be approaching this problem in a fundamentally incorrect way (first time with property testing!)

[saurabhnanda]

I would take a guess that property . hoist somethingToCreateAndRunAppM $ do might work for you. If you have a small example somewhere project somewhere I'd be happy to try and get it working through it with you.

Give me some time to put together a repo with minimal dependencies, but with the essential structure of the actual code.

@jacobstanley please let me know if this is what you had in mind - https://github.com/saurabhnanda/hedgehog-db-testing/blob/master/src/Main.hs

The crux of the problem I'm facing is at https://github.com/saurabhnanda/hedgehog-db-testing/blob/b5720ac6afa184f37b51870800abe0242f0d13c7/src/Main.hs#L184-L196-- everything else just gives context of why I'm writing properties/tests in this manner.

I have commented each function, so it should make sense if you read it from top-to-bottom.

[nhibberd]

Pushed up an example for you - https://github.com/saurabhnanda/hedgehog-db-testing/pull/1

[jacobstanley]

Great! Thanks I'm just having a look now. (edit: Looks like @nhibberd beat me to it :laughing:)

it seems that hedgehog requires a Show instance on all randomly generated values (and therefore it can't generate random functions

Yeah it needs to be able to print what it generated on a failure so that you can figure out what went wrong.

randomChildOfValueA :: (MonadGen m) => m (SavedValueA -> UnsavedChildOfValueA)

^ I think we might be able to make this idea work, will try something with newtypes around that function. The Show instance could use a dummy value for SavedValueA potentially.

[jacobstanley]

Have put up a PR https://github.com/saurabhnanda/hedgehog-db-testing/pull/2 on top of @nhibberd's example which shows how to make a show instance for that randomPost function and also makes the failure output a bit more obvious.

[saurabhnanda]

Thank you so much @jacobstanley and @nhibberd

Let me try both the solutions in my actual codebase and report back.

In parallel:

1. Is my testing strategy alright from a property-testing standpoint? What kind of issues were you referring to when you said the following?

I would recommend not doing any of the DB IO until after the forAll block if at all possible. It can be workable but the way IO works during shrinking can be quite confusing.

2. Would you be open to a PR documenting these solutions? Is there some sort of Hedgehog cookbook that I can contribute back to?

[jacobstanley]

For 1. it's more of an artifact of how hedgehog does shrinking, you could end up with a test run which never creates the user because the ResourceT cleans up but we never run the DB.save which comes after its forAll. Only the code after the last forAll is guaranteed to run every test.

Sometimes this can be OK if you understand what's going on but it's not really worth the trouble imo.

2. Absolutely! I'll give you as many gold stars as you can carry :star: :star: :star:

Yeah some contributors are considering some more substantial documentation like that, not a cookbook exactly, maybe it's worth chatting about it with them. I'll send you a slack invite.

[saurabhnanda]

I've tried the property . hoist (withRollback pool) action trick with a few simple tests in my test-suite and it seems to work absolutely fine. The "pro-tip" about evalM also works well. I've tried to do away with a function that looks like this:

withGen :: (Show a, MonadCatch m)
        => Gen a
        -> (a -> m b)
        -> PropertyT m b
withGen gen action = forAll gen >>= (evalM . lift . action)

-- Usage:
-- withGen (Gen.billingPlan client_) (withThrow . createModel ())

I was expecting the "stack-trace" to get messed up, but surprisingly, hedghog is able to get to the the second line of the stack trace which points to the withGen call-site within my test (the first line is obviously withGen itself). This is amazing 👍

[saurabhnanda]

I'm now stuck with the following:

-- This generates a Wai application that uses the same DB connection, so that
-- I can communicate with the web-app within the same DB txn and e'thing rollsback
-- after the test
withApp :: Pool Connection -> (Wai.Application -> TestM a) -> IO a

myProperty pool = testProperty "my prop" $ property . hoist (withApp pool) $ do ... 

This doesn't type-check due to the type-sig of hoist. I have a possible workaround where the Wai.Application is shoved into the TestM environment, and this structurally becomes the same as the withRollback case. However, is there some other type-fu that can be used to solve this?

[saurabhnanda]

Yeah some contributors are considering some more substantial documentation like that, not a cookbook exactly, maybe it's worth chatting about it with them. I'll send you a slack invite.

@jacobstanley Thank you. I have joined.

[saurabhnanda]

Is there any magic happening in evalM -- whenever I pop it in, my tests seem to violate something at the DB-level. Either the DB txn gets rolled-back early, or it ends-up being shared between two tests. I am getting errors that look like the following...

      ✗ test app failed after 1 test and 247 shrinks.
     -- snip --
            ┃ │ SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"billing_plans\" violates foreign key constraint \"fk_billing_plans_clients\"", sqlErrorDetail = "Key (client_id)=(9463) is not present in table \"clients\".", sqlErrorHint = ""}

... which is otherwise not possible if everything in a single test/property run uses the same DB connection.

What happens during shrinking? Are the IO actions (which are interspersed with random generation) repeated? Could it be that this error message is not the actual test failure, but a new error that occurs during shrinking?

[saurabhnanda]

Okay, I can confirm that evalM and shrinking don't seem to work well together, probably in the presence of IO.

The following two snippets of code, produce completely different error reports

Without evalM

Code:

forAll (Gen.user c) >>= (lift . (const $ Prelude.error "forced"))

Error:

  test app:    FAIL (0.03s)
      ✗ test app failed after 1 test.
      
            ┏━━ test/Test.hs ━━━
        274 ┃ testApp testArgs = testProperty "test app" $ property . hoist (withApp testArgs) $ do
            ┃ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            ┃ │ ━━━ Exception (ErrorCall) ━━━
            ┃ │ forced
            ┃ │ CallStack (from HasCallStack):
            ┃ │   error, called at /Users/saurabhnanda/projects/vl-gitlab/haskell/test/Test.hs:280:44 in main:Test

With evalM

Code:

  forAll (Gen.user c) >>= (evalM . lift . (const $ Prelude.error "forced"))

Error:

  test app:    FAIL (2.16s)
      ✗ test app failed after 1 test and 261 shrinks.

            ┃ │ ━━━ Exception (SqlError) ━━━
            ┃ │ SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"billing_plans\" violates foreign key constraint \"fk_billing_plans_clients\"", sqlErrorDetail = "Key (client_id)=(12909) is not present in table \"clients\".", sqlErrorHint = ""}

[saurabhnanda]

Another update, I don't think evalM is the culprit, though it might be causing the problem to occur earlier / faster.

Right now, when I call failure in a test, and the shrinking process begins, I can see from Postgres logs, that some IO actions (basically DB inserts) are being done again, which result in a completely different error thrown by the DB (due to some DB-triggers). The test-output reports the error thrown during the shrinking process, and not the original error.

How does one deal with something like this?

Underlying cause: We set some session variables in the DB within a transaction (for the purpose of audit logs), which must be reset once the txn is complete. When the shrinking process starts it doesn't know anything about these session variables, and all DB inserts throw an error.

[HuwCampbell]

Shrinking works like backtracking in a non-determinism monad.

~property . hoist (withApp pool) will have some very funky properties because the pool won't run its cleanup until after all shrinks are done.~

You're much better off to do something like:

property $ do
  a <- Gen.something
  b <- Gen.something
  test . hoist (withApp pool) $ do
    all_my_tests
    go_here

That way when shrinking happens, the (withApp pool) will be executed individually for each shrunk case.

Edit: I may be mistaken here. #285 seems to cover off this.

[saurabhnanda]

I'm trying the other approach of making "dependent" generators return unapplied functions, i.e MonadGen m => m (ValueA -> ChildOfValueA).

In parallel, is there any way in which a "hook" can be provided into the shrinking operation? That would allow me to put the DB back into the expected shape for each shrink.

[saurabhnanda]

Both the approaches have problems:

• Doing IO in the middle of random data generation causes unforeseen problems during shrinking. In my case, it causes the shrinking process to completely barf and report errors not related to the original failure!
• The other approach of making the generators return m (ValueA -> ChildValueOfA) doesn't cause problems during shrinking, BUT there is no way to to show/display the actual ChildValueOfA due to which the test fails.

Is there any way around these problems? Perhaps have a hook/callback that allows to control how IO is performed during shrinking?

[saurabhnanda]

Following up on https://github.com/hedgehogqa/haskell-hedgehog/issues/312#issuecomment-515073275 again.

I was preparing for my talk at Functional Conf 2019 [1], which led me to to this blog post about state machine testing in Hedgehog. I never looked at the state-machine testing infra available in Hedgehog due to the extremely complicated types, but that blog most made me look again. And here's what I found in the docs:

Symbolic values: Because hedgehog generates actions in a separate phase before execution, you will sometimes need to refer to the result of a previous action in a generator without knowing the value of the result (e.g., to get the ID of a previously-created user).

And that was one of the problems I was struggling with!

This led me to wonder why state-machine testing wasn't suggested by anyone else in this discussion? Is it still experimental? Or is it too complicated for regular use?

As a follow-up question: are there more walkthroughs of state-machine testing? Even after that blog post and reading the Haddocks, I have no clue how to use this part of the library.

[1] which is mostly based about my learnings as a result of the discussion in this thread

[jacobstanley]

Sorry, I missed this follow up.

I'm happy to see you got enough working for a talk!

This time, I tried with Hedgehog, and got it to work!

It is quite hard to use, I would like to find a better way to do the same thing, but there is another set of posts from QFPL that might help: https://qfpl.io/posts/intro-to-state-machine-testing-3/

Yes, exactly the quote you call out is the really cool thing about state machine testing! I copied the design exactly from the erlang version, but it works out a lot simpler in a dynamic language. There is also quickcheck state machine library that might be worth looking at.

John Hughes gave this great talk on state machine testing that really made me want to have it in Haskell, might be interesting to watch to get a feel for the approach: Testing the Hard Stuff and Staying Sane (he talks about finding bugs in a CAN bus and also race conditions in a database)

I would say what we have in Hedgehog right now is not that much fun to use, but it does work.

[jacobstanley]

I have written about a way to "get the ID of a previously-created user" without using the complicated state-machine testing types that we have:

https://jacobstanley.io/how-to-use-hedgehog-to-test-a-real-world-large-scale-stateful-app/

I think it might even be just as powerful as Hedgehog's state-machine testing!