Ability to set a "reference" job

[michalmuskala]

Right now when benchmarks are compared, they are always compared to the fastest run. I think it would be useful in some situations to set a "reference" run and compare all the other runs to that.

One example would be the benchmark suite of jason, where I'd like to see all the other encoders reported relative to Jason itself, even though jiffy is the fastest most of the time. Right now, I have to do the additional computations manually, to get the desired numbers, that I can use to annoy people convincing them how fast Jason is 😉.

[michalmuskala]

I looked into what could be the best way to do this and I'm not really sure. Right now, it seems, the order is handled by each formatter separately. Should we also move that to be done in just one place at the same time? We'd probably need to pass two things in that case - the ordered scenarios + the base scenario, so the comparison can be easily computed by the formatter.

[PragTob]

@michalmuskala yes that's exactly how I'd like it to be done :) I think right now through the memory vs. runtime split there's a bit of duplication/things that we need to unify or should generalize in general :)