analyzer icon indicating copy to clipboard operation
analyzer copied to clipboard

Automate manual tests

Open sim642 opened this issue 3 years ago • 1 comments

Now that we have cram from #787, it's a good time to use it to automate previously manual tests.

Changes

  1. Add tests for YAML witness generation. The yamlWitnessStrip executable is used to remove non-deterministic metadata parts from the output.
  2. Fix ana.apron.invariant.one-var not filtering enough.
  3. Make update_suite.rb print test names that have no automatic checks. Add NOCRASH, FIXPOINT, NOTIMEOUT and CRAM annotations to suppress that by indicating which implicit aspect is being tested.
  4. Add such suppressing annotations to tests where the reason for no explicit checks is apparent from comments or git history.

TODO

  • [x] #855
  • [ ] Add missing annotations:
    No automatic checks in 01/51 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 02/57 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 03/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 03/22 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 13/62 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 13/65 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 18/22 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 18/23 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 18/24 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 19/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 20/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 20/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 20/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 20/05 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 22/08 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 22/12 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 22/13 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 22/16 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 23/08 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 23/12 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 25/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 27/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 27/07 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 29/25 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 29/26 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 33/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 33/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 33/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 33/04 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 34/05 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 41/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 43/12 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 43/25 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/00 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/13 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/15 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/16 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/17 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/23 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/33 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    
  • [ ] Fix cram test failures on MacOS.

sim642 avatar Oct 06 '22 10:10 sim642

What are our plans vis-à-vis this PR? If we want to have it, I'll be happy to review once it's brought up-to-date with master. Otherwise, I suggest we close it.

michael-schwarz avatar Nov 24 '23 14:11 michael-schwarz

To merge this PR, it would be good to add a white list of tests that already exist. After this PR is merged, old tests can gradually be converted.

jerhard avatar Jul 16 '24 11:07 jerhard