Record/replay mode

[cdsmith]

I would love to extend HMock with some mechanism for running record/replay tests. These are tests that are written as integration tests, but that you want to be able to repeat quickly while refactoring and such without spinning up heavyweight dependencies like a database.

The process goes something like this:

1. You run the code in a the slow integration test setting with real dependencies first. It's run in some mode where HMock gets spliced in so that it sees all of the methods, forwards them to the real implementation, and then sees the response from the real implementation. The integration test makes assertions about the whole system working correctly (NOT worrying about the mocks at all), and either succeeds or fails.
2. If the integration test succeeds, the observed requests and responses are saved into some file format.
3. You can later repeat the integration test, but without the real dependencies and having HMock expect that the actions taken will match its recorded list, and substitute the same responses that happened in the real test.

This raises a whole lot of questions.

• How does HMock record all the arguments, when they aren't necessarily serializable?
• How does HMock record the responses, when they aren't necessarily serializable?
• How does HMock know when the order requirements are important, and when they are not?
• How do you set up such a test to insert HMock into the stack?

A first start would be to just throw runtime errors when intercepting any method without serializable parameter types and responses. You would set up such a test by wrapping your code in something like:

  withRecordReplay options $ do
    codeUnderTest

Here, codeUnderTest would be run in an interceptor monad transformer if you're in record mode, or in MockT if you're in replay mode. In the former case, withRecordReplay would save everything to the transcript. In the latter case, withRecordReplay would automatically load the transcript and set up matching HMock expectations. The choice of mode can be in options, or by looking at command line parameters, or whatever.

As a default, HMock should assert exact orders and parameters and everything. If the file format were human-readable, then it could be modified to relax requirements. But this is probably a bad idea, because you really don't want to have humans edit something that is going to be regenerated and overwritten regularly. Instead, you probably want some way to specify relaxed constraints in options, using Matcher and such. You want to be able to say things like:

• "Any time something matches this matcher, just use this matcher instead of recording its exact values."
• "Never worry about the multiplicity of anything matching this matcher."
• "Do not worry about the exact times on methods matching this matcher."
• "In this entire test, add each unique thread id's calls to a different sequential expectation, and don't assert anything about cross-thread ordering of methods."