benchexec
benchexec copied to clipboard
score calculation for SV-COMP 2021 should take the result of the new witness linter into acount
Last year we had some problems with the score calculation because a validator falsely flagged witnesses as invalid. As a result, the code that prevents a witness from counting as validated was removed from mergeBenchmarkSets.py via the following commit:
https://github.com/sosy-lab/benchexec/commit/f34ee4503f3698e97090b89ff6bfc6e2040b4dc6
The idea came up to have a dedicated tool that syntactically checks the witnesses and decides whether these are syntactically valid or not. The tool currently resides on the branch https://github.com/sosy-lab/sv-witnesses/tree/witnessLinter
For SV-COMP 2021, the score computation should depend on the decision of the witness linter to decide whether a witness is syntactically invalid and therefore should not count as validated.
This will depend on the tool-info module for the witness linter, which is not yet written.