we should capture some stats on the box that cmp was run on.

[notbenh]

it's nice to have local comparisons, but if we start to do run-to-run comparisons then we should have a way to start to standardize things so that we don't skew results to favor those run on faster boxes.

[trenton]

The compile-step duration should be excluded for compiled languages (Java, Scala)

[notbenh]

Exactly, leto and I have been poking at a rebuild of the internals for bench's benchmarker (cleverly called toolbench at this point) that currently supports both a startup and cleanup step that are untimed. I would say that were about 90% done with a first working copy, then it would be a small bit of work to mate the two up so that bench really just becomes an euler_bench config and dir structure processor that hands off the correct syntax to toolbench.

If you want to follow along the code is here: http://github.com/notbenh/tool_bench