Functional test framework

[jot2re]

We need a generic, easy to use, framework for setting up functional tests of protocols. Right now it seems to be individual for each protocol with a lot of code-copy. Basically the TestRuntime from the Mascot package is very nice, but tightly bounded to Mascot. I think the best would be to just generalise this.

[pffrandsen]

I agree the current system is inadequate. I think though it requires a bit more thought - and as nice as the TestRuntime is, I think it has it shortcomings.

There are several types of tests, the two biggest parts are:

• Protocol suites (e.g. TestDummyArithmeticProtocolSuite) which should be a test suite created with a supplier of a (numeric) running protocol suite
• Components which could be mascot, in which case you need a par-function - similar, but not identical to, runPerPartyTasks

A better framework would include caching of engines and allow parallel execution of tests.

I think in order to understand this issue, we should map out the needs of the persons using the tests, i.e. expand on my list and present examples.

[jot2re]

Some nice features that could be added to this hypothetical framework:

• Easily running methods in parallel to ensure that parallel execution does in fact work.
• A nice way of verifying exceptions are thrown, and are of the type and with message expected.
• Easy methods for typical sanity checks (such as different from 0 checking and uniqueness).
• Perhaps a way to test that things sent that are supposed to look random have approximately an equal amount of 0 and 1 bits. (Or some other easy randomness metric).
• Perhaps a way to ensure that if a party sends something incorrect, then the result is either as expected or a Malicious Exception is thrown