criterion icon indicating copy to clipboard operation
criterion copied to clipboard

Add documentation regarding how Criterion does its measurements

Open paulolieuthier opened this issue 9 years ago • 2 comments

This is really important for people writing papers about Haskell who use Criterion to perform measurements. I'm helping a team that is using a modified version of Criterion to measure the energy consuption of benchmarks (we plan to file a pull request when it's done).

It's mainly to explain the statistical reasoning behind the runBenchmark function from Criterion.Measurement.

paulolieuthier avatar Jan 16 '16 16:01 paulolieuthier

That sounds great. If you have suggestions or a PR, they would be welcome. We're trying to make a new wave of changes to this library, so now is a good time.

At the very least we could link some of the good blog posts that explain the linear regression methodology.

One thing I would like to clarify to users is the difference between taking the "Mean" and taking the "OLS regression". I have found that people are often not clear on that and it's not self explanatory.

Likewise the "y" and "iters" labels are hard to remember and not super accessible.

rrnewton avatar Jun 08 '16 19:06 rrnewton

This is really important for people writing papers about Haskell who use Criterion to perform measurements.

+💯

One thing I would like to clarify to users is the difference between taking the "Mean" and taking the "OLS regression". I have found that people are often not clear on that and it's not self explanatory.

Also + 💯 . I have some clue about linear regression, but that makes sense if I'm plotting a dependent variable against an independent variable—which are the variables here? The dependent one should be runtime, but what's the independent variable?

Never got a clue until today. #122 suggests it's the number of iterations in a "single" measurement, something I've never seen in benchmarking frameworks. (Also, the paper cited in #122 seems to question the approach).

Blaisorblade avatar Sep 28 '16 15:09 Blaisorblade