benchexec
benchexec copied to clipboard
sosy-lab
BenchExec: A Framework for Reliable Benchmarking and Resource Measurement
Addresses #830 Fixed the display on initial render by defining a function named `getNewFileteredColumnValues`. It returns the new Filtered Column Value and is subsequently used in `useEffect` to display changes...
Factored out from #994: In use cases like QComp the existing ways to define an expected verdict and classify the actual result into correct/wrong are not enough, because for example...
Based on discussion: I propose to allow to attach arbitrary tags to properties in task-definitions: ```yml properties: - property_file: ../properties/unreach-call.prp expected_verdict: true tags: - fast - ci - property_file: ../properties/valid-memsafety.prp...
The HTML table from `table-generator` on the Summary tab currently only has counts for correct/correct-unconfirmed/incorrect results, separately for true and false. Since there can be a lot more different statuses,...
The CSV tables exported by table-generator have a layout that is inspired by the HTML tables, but this sometimes makes them hard to use programmatically in other tools. We should...
I want to provide some improvement suggestions to the documentation after understanding more about benchexec. In particular, add a clearer overview of what which tool is doing, how it can...
As discussed [previously](https://github.com/sosy-lab/benchexec/issues/717) we will remove support for Python 3.7 in one of the next releases. I suspect this will not be critical because all currently supported versions of Ubuntu...
Factored out from #994: Right now, BenchExec creates up to two derived values from the result of the tool: the result category (correct/wrong/etc.) and a score. Once we support more...
Addresses #989 The container implementation is easy. Just pass the pid we already got to `parent_setup_fn`. They already wait for the parent to signal readiness. However, the non-conainer implementation does...
In table-generator, we use the rounding mode `ROUND_HALF_UP`. We set this globally in `columns.py`, which creates problems if someone imports table-generator, and for #917. We also do not want to...