benchexec

benchexec copied to clipboard

Update test framework

8

Currently the tests are executed with `nose`, but this framework is dead. We should switch to a new framework, for example `nose2` or `py.test`. It seems `py.test` has quite useful...

PhilippWendler

We have [some documentation](https://github.com/sosy-lab/benchexec/blob/main/doc/INSTALL.md#setting-up-cgroups-in-a-docker-container) for using BenchExec in a Docker container. Podman is a more modern alternative, and it might not have the problems that Docker containers have. So we...

PhilippWendler

Improve handling of nested mounts

1

If we create nested containers (e.g., with `containerexec`) and use overlayfs for some directory in both containers, it looks like the inner `containerexec` mounts two instances of overlayfs for the...

PhilippWendler

BenchExec detects out-of-memory situations that occur because a benchmarked process violates the configured memory limit, and sets the appropriate termination reason in its output. However, we do not detect cases...

PhilippWendler

Status drop-down does not correctly display on initial render

1

When opening the table tab of our HTML tables with a defined status/category filter, e.g., via [this link](https://sosy-lab.github.io/benchexec/example-table/svcomp-simple-cbmc-cpachecker.table.html#/table?filter=0(0*status*(category(in(correct)))))), the drop-down filter does not correctly display the defined filter, it displays...

PhilippWendler

Value too large for defined data type when creating a file

2

Whenever I create a file from within the `runexec` container (doesn't matter the command; `touch`, `bash -c`, etc.), it gives me this error. Minimum non-working example: ``` $ runexec --read-only...

charmoniumQ

The documentation on the use of properties files and how to indicate whether task results are correct can be improved

4

Hello! I haven't been able to figure out how to get my results included in the "Correct only" results in the generated tables. This is despite having spent quite some...

shonfeder

Output/UI should not rely on colors

1

Right now, when running benchexec, the output of each verdict ("true" or "false") is shown in green or red in order to indicate whether this is a true or false...

cyrille-artho

Output files should be made world-readable

1

For our P4 extension of benchexec (in contrib), we need to run all tests as root, because a non-root user cannot create virtual network interfaces. At the end of the...

cyrille-artho

cgroupsv2: initial support

4

This is an initial implementation to support cgroups v2 and v1 in parallel. This should work out of the box if pystemd is installed. Otherwise to run benchexec/runexec on a...

globin