deepstate
deepstate copied to clipboard
Add features to native test harness
Besides reading input from a directory as described in #3, how else can we enhance the native test harness? What is actually worth implementing?
Ideas:
- Report discrepant runs of test input that has an implied expected outcome. For example, when replaying a test case with a suffix like
.pass, but a failure is observed, flag and report to the user. - Permit passing individual test cases files directly, with desired test name, rather than just accepting an input directory.
- Consider other kinds of structured test run reporting.
- Add a flag that only runs tests whose name matches an expression.
In the cast of discrepant runs, I think we may want to save them as .flaky files, or something like this. This is a good catch, as it could indicate an error in a symbolic backend, or a non-deterministic test. If possible, we'd probably want to rename the existing test case file from either .crash or .pass to .flaky.
Other options sound good. It might be possible to use the C++ regular expression library to provide these advanced test selection features.