analyzer
analyzer copied to clipboard
Automate manual tests
Now that we have cram from #787, it's a good time to use it to automate previously manual tests.
Changes
- Add tests for YAML witness generation. The
yamlWitnessStripexecutable is used to remove non-deterministic metadata parts from the output. - Fix
ana.apron.invariant.one-varnot filtering enough. - Make
update_suite.rbprint test names that have no automatic checks. AddNOCRASH,FIXPOINT,NOTIMEOUTandCRAMannotations to suppress that by indicating which implicit aspect is being tested. - Add such suppressing annotations to tests where the reason for no explicit checks is apparent from comments or git history.
TODO
- [x] #855
- [ ] Add missing annotations:
No automatic checks in 01/51 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 02/57 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 03/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 03/22 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 13/62 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 13/65 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 18/22 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 18/23 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 18/24 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 19/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 20/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 20/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 20/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 20/05 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 22/08 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 22/12 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 22/13 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 22/16 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 23/08 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 23/12 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 25/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 27/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 27/07 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 29/25 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 29/26 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 33/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 33/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 33/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 33/04 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 34/05 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 41/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 43/12 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 43/25 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 44/00 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 44/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 44/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 44/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 44/13 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 44/15 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 44/16 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 44/17 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 44/23 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) No automatic checks in 44/33 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?) - [ ] Fix cram test failures on MacOS.
What are our plans vis-à-vis this PR? If we want to have it, I'll be happy to review once it's brought up-to-date with master. Otherwise, I suggest we close it.
To merge this PR, it would be good to add a white list of tests that already exist. After this PR is merged, old tests can gradually be converted.