benchexec
benchexec copied to clipboard
sosy-lab
BenchExec: A Framework for Reliable Benchmarking and Resource Measurement
Currently `benchexec --commit` requires that the repository has no changes (untracked files are ok), otherwise it just does nothing. It should be enhanced to always commit the result files without...
For inspection of solver output, it could be useful to run the stdout lines through a filter that adds time stamps as prefix, like in syslog. Problem: Some verifiers do...
I'm not sure if this idea fits within the direction you guys would like to take BenchExec, however, I like to get my ideas written down because I forget them...
To increase confidence in time results, it may sometimes be useful to run the command multiple times, and report all results, as well as mean/median/standard deviation etc. This could be...
Similarly to time limits, BenchExec should support a "soft" memory limit. When it is reached and the kernel notifies us, we should send the TERM signal to the process, increase...
Provide a way to have "derived" columns in tables, i.e. columns with values that are computed from other column values. First step would be to be able to use only...
It would be really nice to be able to oversubscribe memory. I realize this causes results to be unreliable, however, in preparation for svcomp it would be nice to more...
There is a [paper](https://dl.acm.org/citation.cfm?id=1508275) that shows that the size of the environment variables of a process can have a non-predictable impact on the performance. In order to avoid this and...
Currently, if required files are missing, only a warning is printed: https://github.com/sosy-lab/benchexec/blob/2b5d1fb2d1a806da397af528e79412a74aa2cda1/benchexec/model.py#L400-L404 But the tool is executed nevertheless. Is there any reason to still execute the tool in such cases?...
Same as #840, but for containerexec, for which this is more difficult to solve. Cgroups provide the possibility to freeze processes, which prevents them from being killed (even with ´kill...