sv-benchmarks
sv-benchmarks copied to clipboard
sosy-lab
[WIP] CI is too slow
WORK IN PROGRESS. This issue discusses the problem with the slow CI and possible solutions.
Use CCache for GCC and Clang
I tried to add ccache for the gcc and clang job. This does not bring any improvement, because we use the compiler only for the first step of the compilation (i.e., no linking), which does sadly not benefit from ccache. To test this on your own system, install ccache and use make CC='ccache gcc-5' for executing the CI-job.
Speedup the script compare.sh (see #879)
This script requires the most runtime and is mostly the reason for timeouts. It is written in BASH and is horribly inefficient in some parts. I will replace it with a faster Python-based implementation, see #879.
Split the benchmark set (at least for CI)
Another obvious possibility to improve the CI checks would be to generate more parallel CI jobs. As most categories are independent, this should be easy to do. The drawback is the maintainance cost, i.e., when renaming categories, directories, files. However, if the repository grows more and more, this will be the only scalable solution. With #821 this was partially done, i.e., the CI checks for one large set of new benchmarks are executed in a separate (parallel) CI job.
Exclude tasks from being checked again and again
For unchanged tasks we could store a hash and them on repeated checking. This would have some maintainance costs when changing makefile rules, compiler options, and with compiler updates.
Change to a faster CI hardware
In the future we might fully migrate to the GitlabCI, where we can include our own hardware. This might improve the runtime.
Given that most PRs only touch a (small) subset of tasks, I think the most impactful and (simple) optimization is to avoid checking unchanged tasks. Isn't it already enough to exclude all tasks that are not present in git diff?
No, that is not sufficient, but a good heuristic. Some files might include others or depend on them. Sometimes, we change global scripts, options like machinemodel, or the compiler.
Currently, all CI jobs run as expected. We will take a detailed look again as soon as there are more serious problems. @hoangmle If you have a concrete solution, then feel free to provide a pull request for that.
Yes, that should be implemented as a top-level heuristic. If a PR changes *.i and *.c only (most PRs do), then you are safe, otherwise just run a full CI job.
I could give it a try, but since it is not top priority anymore, that might not happen soon.
Do you know a way to determine which files a PR touches? Note that a PR can have an arbitrarily long list of commits.
I guess if $TRAVIS_PULL_REQUEST then git diff $TRAVIS_BRANCH --name-only would do the job, to also support push builds, we would need $TRAVIS_COMMIT_RANGE (these enviroment variables are defined here https://docs.travis-ci.com/user/environment-variables/#convenience-variables)