Hashes of all input that BenchExec gets

[dbeyer]

Towards 100 % replicability, it would be great to have BenchExec store an SHA256 hash for each input file that it touches.

Use case: A user inspects slightly outdated result tables generated from BenchExec results (e.g., provided by a competition organizer) and gets puzzled why the result differs from what the user gets on the own execution. A comparison of the hashes of the input file would quickly reveal that the user is actually looking at a very different experiment.

This use case happens often during times of high frequency of changes to the benchmark set that BenchExec is operating on (e.g., during training phases of competitions such as SV-COMP and Test-Comp).

As a solution to this problem, each BenchExec result needs to to stored together with separate information that identifies the exact version of the input files.

The tag already stores the input file name, but obviously, this does not identify the input. Example:

  <run files="[../sv-benchmarks/c/bitvector-loops/verisec_sendmail_tTflag_arr_one_loop.i]" name="../sv-benchmarks/c/bitvector-loops/verisec_sendmail_tTflag_arr_one_loop.yml" properties="coverage-error-call">

How about adding

fileHashes="[df33a606ed8dff6e0cd458b6a2a86f46509c229534b5b2e3fec472f3ad3ab62c]"

to make it clear which file the run was about?

Would the computation time be an issue? I would argue no, because it it just a fraction of the overall execution time of BenchExec.

[dbeyer]

Instead of fileHashes I now propose to use an swhid using Python module https://pypi.org/project/swh.model/.

This also works for directories/sets of files.

[PhilippWendler]

Hm, unfortunately there are several problems with this module, the most important that it is under GPL, and so we cannot really use it in BenchExec.

I googled and found a spec for these SWH ids, so in principle we could reimplement this.

I haven't heard of these ids before, but it seems they are becoming more common. I also do not know any alternative standard that we could use for this, so it might be the most common standard for what we need.

What makes me skeptical, though, is their choice of SHA-1. There are known collisions of this hash function and git is also moving away from it. Basing a new project on this does not seem like a good idea.