nvbench icon indicating copy to clipboard operation
nvbench copied to clipboard

Add option for reporting median runtime

Open sleeepyjack opened this issue 2 years ago • 1 comments

I stumbled across this Slack thread recently when I was trying to measure small kernels with nvbench and got fluctuating results. As @senior-zero notes in the thread, the variance of the minimum runtime is usually smaller than that of the average runtime.

So it would be very useful if nvbench would provide a way to generate these values directly. We could also report additional metrics, such as max or median runtime.

sleeepyjack avatar Jun 28 '23 15:06 sleeepyjack

min/max were added in #204. They are hidden by default, but there's an example in that PR that shows how to display them in the markdown output.

We don't compute the median, is there a strong usecase for this? It'd be somewhat expensive to compute as we don't currently sort the sample timings and I'd prefer to avoid that if possible.

alliepiper avatar Apr 30 '25 14:04 alliepiper