API compatibility testing wish list

[lemmih]

Issue summary

Assorted features that would be nice to have:

• [x] Filtering based on method name. Similar to how tests can be filtered when running cargo nextest or cargo test.
• [ ] Option for generating exhaustive tests based on snapshots. Right now, we only tests a small subset of the possible queries. With exhaustive testing, we would generate a Filecoin.ChainGetTipSetByHeight call for each available epoch (calibnet is currently at around 1 million epochs). Same goes for Filecoin.ChainGetBlock, Filecoin.ChainGetMessage, Filecoin.StateGetActor, Filecoin.StateMinerPower, etc. These calls have a finite set of input arguments and we can exhaustively test all of them.
• [ ] Option for sampling tests. Running millions or tens of millions of tests is time consuming. It would be nice to run, say, 100 randomly sampled tests for each RPC method. This is similar to what quickcheck does.
• [ ] Run tests in parallel.

Other information and links