benchexec icon indicating copy to clipboard operation
benchexec copied to clipboard

score calculation for SV-COMP 2021 should take the result of the new witness linter into acount

Open MartinSpiessl opened this issue 5 years ago • 0 comments
trafficstars

Last year we had some problems with the score calculation because a validator falsely flagged witnesses as invalid. As a result, the code that prevents a witness from counting as validated was removed from mergeBenchmarkSets.py via the following commit:

https://github.com/sosy-lab/benchexec/commit/f34ee4503f3698e97090b89ff6bfc6e2040b4dc6

The idea came up to have a dedicated tool that syntactically checks the witnesses and decides whether these are syntactically valid or not. The tool currently resides on the branch https://github.com/sosy-lab/sv-witnesses/tree/witnessLinter

For SV-COMP 2021, the score computation should depend on the decision of the witness linter to decide whether a witness is syntactically invalid and therefore should not count as validated.

This will depend on the tool-info module for the witness linter, which is not yet written.

MartinSpiessl avatar Sep 19 '20 22:09 MartinSpiessl