sel4bench icon indicating copy to clipboard operation
sel4bench copied to clipboard

CI actions should build with clang also

Open axel-h opened this issue 3 years ago • 4 comments
trafficstars

CI actions should build with clang also besides just gcc and benchmaks should run with both binaries. Building with clang is expected to improve code quality, because clang raises more warnings. Large discrepancies in the benchmark results can indicate that the benchmark either has a strong compiler dependency or its implementation has issues.

Building with clang also seems rather trivial, running both binaries will be more work.

axel-h avatar Feb 12 '22 14:02 axel-h

That does sound reasonable. @kent-mcleod any concerns on adding a build and run with clang for all benchmarks? It'll take more machine time, but it could be informative.

lsf37 avatar Feb 12 '22 21:02 lsf37

Then again, the x86 runs each take 50 min, and if we want comparable numbers they need to run on the same machine, so we would be blowing the time out to 100 min.

lsf37 avatar Feb 12 '22 21:02 lsf37

If the concern is about hardware resources, it could be a topic for the budget planning.

axel-h avatar Feb 12 '22 22:02 axel-h

If we want gcc/clang to be comparable, should it not be the exact same machine? In that case it doesn't matter if we buy more. We do have more x86 boards, but I'm not sure they are the same configurations. @wom-bat might be able to confirm. x86 machines should be easy to add, I think, if that helps.

Another way to look at this would be to see if all x86 benchmarks are actually needed/informative. What currently exists is what has accumulated and not necessarily designed to be the optimal set for CI.

lsf37 avatar Feb 12 '22 23:02 lsf37