google
Git Checkout might not work if there are multiple copies of a repo.
The benchmark.yaml file has a checkout hash that can be used to checkout a specific commit before running the benchmarks. We have observed that the checkout seems to fail spuriously in some benchmarks, like in libjpeg-turbo_libjpeg_turbo_fuzzer.
By looking at the source code, we found out that the code seems to go into each directory and try to checkout the specific commit, stopping at the first folder that succeeded.
This means that, in the case of libjpeg-turbo that has at least 3 folders (one for each branch), only one of them will be in the right commit. Furthermore, it seems like the order of the directory iteration is not stable, so it might change between runs.
We haven't looked to see which other benchmarks are affected.
Let's remove the autocheckout and checkout the repo in the dockerfile
