nonius icon indicating copy to clipboard operation
nonius copied to clipboard

HTML report shows wrong data with multiple param values

Open jtippet opened this issue 5 years ago • 0 comments

Create a param named X and run a test with multiple param values, like benchmark --reporter=html --param=X:+:0:1:10. The HTML report will show a nice summary page that shows a line plot of how the benchmarks fare on each param value.

Unfortunately, the data is just wrong. The data is correct only for the first (leftmost) column; all other columns have the data mixed up between the various series.

Here's an example: image

Note how the orange and blue series seem to trade off on looking bad. That's not backed up by the raw data -- in the raw data, the orange series should always be bad, and the blue should always be okay.

A corrected graph would look like this: image

jtippet avatar Feb 25 '20 23:02 jtippet